Add openvino export configs and support chatglm #454

eaidova · 2023-10-16T15:33:35Z

What does this PR do?

Introducing export configs for openvino. ChatGLM2 model used as example for demonstration that this API extension work

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you make sure to update the documentation with your changes?
Did you write any new necessary tests?

HuggingFaceDocBuilderDev · 2023-10-16T15:46:57Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

echarlaix · 2023-10-17T17:17:59Z

optimum/exporters/openvino/dummy_input_generators.py

+    }
+
+
+class ChatGLM2DummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):


I think we can move it to optimum directly as it's not specific to OpenVINO

https://github.com/huggingface/optimum/blob/e7bd60dd2c1e295263ba57a4e468a62ab5b179e8/optimum/utils/input_generators.py#L318

echarlaix · 2023-10-17T17:19:36Z

optimum/exporters/openvino/normalized_configs.py

+
+
+@register_normalized_config("chatglm")
+class ChatGLM2NormalizedConfig(NormalizedTextConfig):


same we could move it to optimum

https://github.com/huggingface/optimum/blob/e7bd60dd2c1e295263ba57a4e468a62ab5b179e8/optimum/utils/normalized_config.py#L80

echarlaix · 2023-10-17T17:29:34Z

optimum/exporters/openvino/__main__.py

+            )
+        else:
+            onnx_config_constructor = TasksManager.get_exporter_config_constructor(
+                model=model, exporter="openvino", task=task


here I would prefer we keep the onnx config so that we don't duplicate code as I'm unsure why we would need it for now, is there a specific reason for that @eaidova ?

the main idea of this pr to allow override configs for onnx for openvino.
I see 2 reasons for that:

Adding configurations for supporting new models now requires to guiranree that model can be exported in onnx (and possibly even provide pipeline for running it with ORT). However, we do not use export to onnx as default path for export models to openvino anymore. The set of supported operations from torch in openvino can be different from set of ops supported to torch-onnx export. For example, we successfully convert Tapas model family to openvino (for example https://huggingface.co/google/tapas-tiny-finetuned-wtq), but onnx export failed with unsupported aten::scatter_reduce

Unblocking different optimizations of existing models that specific for openvino. For the same reasons that we do not export models to onnx and has own inference pipelines, in some time it may happens that onnx and openvino export paths can become different and require different configurations (for example if I want to merge 2 decoders in seq2seq models in the way how it will be convenient for openvino or in some other our plans we want to try export caulsallm models with including beam search inside models). Some simple example, where onnx configuration is not perfectly fit for us, text-generation-with-past, if I understand, for onnx default path is a merged model, while we use model-with-past. ONNX configs for a model with past fill input_ids with only 1 token for this case lead to some exporting model troubles, that require models patching and consider each new case as separated (in the latest optimum, created patchers per model, now there is just function that check model type and apply the patch on pytorch model), but it can be avoided in majority cases if input_ids will have 2 tokens instead of 1. Also it uses only dynamic batch and static sequence len (=1) for this case, that lead to extra model reshaping before loading to make sequence len dynamic for this input.

Simplification of new models enabling flow. We really like optimum and all its features for its smooth user experience (thank you very much for everything what you do) and recommend it to our openvino users as the main path for running inference for HF models using openvino, so we are very interested in extension of supported models and having the latest trading models available running with openvino. Support of cli and API for export models open the door for converting everything that supported in optimum directly in openvino (even if we do not have some OVModelForXXX classes) directly to openvino and it is the great step for us. But now, it is not enough to just install optimum-intel from git for get the latest available models from optimum side, it requires also install optimum for git. But there is no guarantee that they are synchronized. Like you already highlighted about changes with position_ids for example or another thing what I recently found trying to run mistral model that now for some models it is not enough to specify only with_past=True for getting the model with past in inputs and outputs in the same time, additional with_past_in_inputs flag should be passed in config. We need to wait the next official realse for aligning and getting new models supported that maybe non-convinient for us.

We do not duplicate code, (for majority cases configs mapping filled in runtime, just reusing the same onnx config, but if we have own one for openvino specifc, we will use own) just allow overriding some export configs if it is applicable.

echarlaix · 2023-10-17T17:30:14Z

optimum/intel/openvino/modeling_decoder.py

        return {
            "input_ids": input_ids,
            "past_key_values": past_key_values,
            "use_cache": self.use_cache,
-            "position_ids": None,
-            "attention_mask": kwargs.get("attention_mask", None),
+            "position_ids": position_ids,


thnaks for adding the position_ids support added in huggingface/optimum#1381

echarlaix · 2024-01-21T18:49:10Z

optimum/exporters/openvino/model_configs.py

+def create_register(overwrite_existing: bool = False):
+    def wrapper(model_type: str, *supported_tasks: str) -> Callable[[Type], Type]:
+        def decorator(config_cls: Type) -> Type:
+            mapping = TasksManager._SUPPORTED_MODEL_TYPE.get(model_type, {})
+            mapping_backend = mapping.get("openvino", {})
+            for task in supported_tasks:
+                normalized_task = task
+                if "-with-past" in task:
+                    normalized_task = task.split("-with-past")[0]
+                if normalized_task not in TasksManager.get_all_tasks():
+                    known_tasks = ", ".join(TasksManager.get_all_tasks())
+                    raise ValueError(
+                        f'The TasksManager does not know the task called "{task}", known tasks: {known_tasks}.'
+                    )
+                if not overwrite_existing and task in mapping_backend:
+                    continue
+                mapping_backend[task] = make_backend_config_constructor_for_task(config_cls, task)
+            mapping["openvino"] = mapping_backend
+            TasksManager._SUPPORTED_MODEL_TYPE[model_type] = mapping
+            return config_cls
+
+        return decorator
+
+    return wrapper
+
+
+register_in_tasks_manager = create_register(True)


could it simplified with :

Suggested change

def create_register(overwrite_existing: bool = False):

def wrapper(model_type: str, *supported_tasks: str) -> Callable[[Type], Type]:

def decorator(config_cls: Type) -> Type:

mapping = TasksManager._SUPPORTED_MODEL_TYPE.get(model_type, {})

mapping_backend = mapping.get("openvino", {})

for task in supported_tasks:

normalized_task = task

if "-with-past" in task:

normalized_task = task.split("-with-past")[0]

if normalized_task not in TasksManager.get_all_tasks():

known_tasks = ", ".join(TasksManager.get_all_tasks())

raise ValueError(

f'The TasksManager does not know the task called "{task}", known tasks: {known_tasks}.'

)

if not overwrite_existing and task in mapping_backend:

continue

mapping_backend[task] = make_backend_config_constructor_for_task(config_cls, task)

mapping["openvino"] = mapping_backend

TasksManager._SUPPORTED_MODEL_TYPE[model_type] = mapping

return config_cls

return decorator

return wrapper

register_in_tasks_manager = create_register(True)

register_in_tasks_manager = TasksManager.create_register("openvino")

echarlaix · 2024-01-21T19:08:17Z

optimum/exporters/openvino/model_configs.py

+
+@register_in_tasks_manager("chatglm", *["text-generation", "text-generation-with-past"])
+class ChatGLM2OpenVINOConfig(TextDecoderOnnxConfig):
+    NORMALIZED_CONFIG_CLASS = ChatGLM2NormalizedConfig


not sure we need ChatGLM2NormalizedConfig

Suggested change

NORMALIZED_CONFIG_CLASS = ChatGLM2NormalizedConfig

NORMALIZED_CONFIG_CLASS = NormalizedConfig.with_args(

vocab_size="padded_vocab_size",

num_layers="num_layers",

)

echarlaix · 2024-01-21T19:27:14Z

optimum/exporters/openvino/dummy_input_generators.py

+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+


can be moved to optimum/intel/utils/input_generators.py

eaidova · 2024-02-21T08:29:59Z

closed in flawor #568

Add openvino export configs and support chatglm

e57baac

eaidova force-pushed the ea/export_config branch from ce9d0cb to e57baac Compare October 16, 2023 15:39

AlexKoff88 requested a review from echarlaix October 16, 2023 15:46

echarlaix reviewed Oct 17, 2023

View reviewed changes

eaidova added 2 commits October 19, 2023 10:20

copyrights and code fixes

fae7802

enable attention mask and fix accuracy issue for chatglm

e596cc7

eaidova force-pushed the ea/export_config branch from ce6cdaf to e596cc7 Compare November 3, 2023 11:09

echarlaix reviewed Jan 21, 2024

View reviewed changes

eaidova closed this Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add openvino export configs and support chatglm #454

Add openvino export configs and support chatglm #454

eaidova commented Oct 16, 2023

HuggingFaceDocBuilderDev commented Oct 16, 2023

echarlaix Oct 17, 2023

echarlaix Oct 17, 2023

echarlaix Oct 17, 2023

eaidova Oct 18, 2023 •

edited

Loading

echarlaix Oct 17, 2023

echarlaix Jan 21, 2024

echarlaix Jan 21, 2024

echarlaix Jan 21, 2024

eaidova commented Feb 21, 2024

		}


		class ChatGLM2DummyPastKeyValuesGenerator(DummyPastKeyValuesGenerator):



		@register_normalized_config("chatglm")
		class ChatGLM2NormalizedConfig(NormalizedTextConfig):

-    NORMALIZED_CONFIG_CLASS = ChatGLM2NormalizedConfig
+    NORMALIZED_CONFIG_CLASS = NormalizedConfig.with_args(
+        vocab_size="padded_vocab_size",
+        num_layers="num_layers",
+    )

Add openvino export configs and support chatglm #454

Add openvino export configs and support chatglm #454

Conversation

eaidova commented Oct 16, 2023

What does this PR do?

Before submitting

HuggingFaceDocBuilderDev commented Oct 16, 2023

echarlaix Oct 17, 2023

Choose a reason for hiding this comment

echarlaix Oct 17, 2023

Choose a reason for hiding this comment

echarlaix Oct 17, 2023

Choose a reason for hiding this comment

eaidova Oct 18, 2023 • edited Loading

Choose a reason for hiding this comment

echarlaix Oct 17, 2023

Choose a reason for hiding this comment

echarlaix Jan 21, 2024

Choose a reason for hiding this comment

echarlaix Jan 21, 2024

Choose a reason for hiding this comment

echarlaix Jan 21, 2024

Choose a reason for hiding this comment

eaidova commented Feb 21, 2024

eaidova Oct 18, 2023 •

edited

Loading