support stateful model #493

eaidova · 2023-12-21T06:08:59Z

What does this PR do?

continue work on stateful models support based on #486

TO DO

Testing on a wider range of topologies
Type hints and doc strings
Stateful models by default #486 (comment)
Stateful models by default #486 (comment)
Stateful models by default #486 (comment)

…is not supported

HuggingFaceDocBuilderDev · 2023-12-21T06:14:19Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

eaidova · 2023-12-21T14:10:27Z

@echarlaix @AlexKoff88 please take a look, while @slyalin on vacation, I finished his PR with stateful models support

optimum/exporters/openvino/stateful.py

optimum/exporters/openvino/better_transformer_patch.py

tests/openvino/test_modeling.py

slyalin · 2024-01-12T11:05:02Z

@echarlaix, we are ready to merge.

optimum/exporters/openvino/convert.py

optimum/commands/export/openvino.py

Co-authored-by: Sergey Lyalin <[email protected]>

helena-intel

I have a few typo fixes and small comments but none of them should block merging. We should also add this to documentation, but there is more documentation to be fixed before 2023.3 so that can be done in a separate PR. Thanks for the PR @eaidova !

optimum/exporters/openvino/model_patcher.py

optimum/exporters/openvino/stateful.py

optimum/commands/export/openvino.py

optimum/exporters/openvino/convert.py

helena-intel · 2024-01-12T16:47:04Z

optimum/exporters/openvino/convert.py

+                # TODO: Implement stateful for ONNX path as well, not doing it right now because of lack of validation
+                logger.warn(
+                    "[ WARNING ] Making stateful models is not supported when exporting to ONNX as an intermediate step. A stateless model will be exported instead. It may result in sub-optimal inference performance."
+                    "Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path."


Suggested change

"Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path."

"Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path."

I think most people will need to export the model they are exporting and "provide another model" is then not really helpful. In my experience, when export with PyTorch failed and ONNX was used instead, it was usually because of a technical issue, with optimum-intel or dependencies. So we could suggest to upgrade dependencies with pip install --upgrade --upgrade-strategy eager optimum[openvino] and maybe suggest to create an issue if exporting with PyTorch still fails?

Co-authored-by: Helena Kloosterman <[email protected]>

optimum/exporters/openvino/stateful.py

optimum/exporters/openvino/__main__.py

optimum/commands/export/openvino.py

optimum/exporters/openvino/__main__.py

optimum/exporters/openvino/stateful.py

tests/openvino/test_modeling.py

tests/openvino/test_quantization.py

tests/openvino/test_modeling.py

Co-authored-by: Ella Charlaix <[email protected]>

AlexKoff88 · 2024-01-16T05:32:52Z

@echarlaix, I think we are ready to merge

echarlaix

Looks good thanks for the addition, can you confirm all tests are passing with openvino-nightly (or another version enabling stateful models) before we can merge ?

optimum/exporters/openvino/__main__.py

Co-authored-by: Ella Charlaix <[email protected]>

eaidova · 2024-01-16T10:51:15Z

Looks good thanks for the addition, can you confirm all tests are passing with openvino-nightly (or another version enabling stateful models) before we can merge ?

I can confirm, I test this locally on my working machine

helena-intel · 2024-01-16T12:20:01Z

Looks good thanks for the addition, can you confirm all tests are passing with openvino-nightly (or another version enabling stateful models) before we can merge ?

I can confirm, I test this locally on my working machine

@eaidova Do the tests that fail with nightly here also pass for you locally with nightly/2023.3? https://github.com/huggingface/optimum-intel/actions/runs/7473712623/job/20526199806

FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_0_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_1_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_2_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_3_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_00_text_generation - AssertionError: 44 != 46
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_01_text_generation_with_past - AssertionError: 44 != 46
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_09_fill_mask - AssertionError: 68 != 70
FAILED tests/openvino/test_quantization.py::OVQuantizerTest::test_ovmodel_static_quantization_1 - AssertionError: 22 != 23
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_automodel_weight_compression_1 - AssertionError: 45 != 44
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_4bit_weight_compression_0 - AssertionError: 295 != 302
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_8bit_weight_compression_1 - AssertionError: 44 != 46
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_load_with_compressed_weights_00 - AssertionError: 44 != 46
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_load_with_compressed_weights_01 - AssertionError: 68 != 70

eaidova · 2024-01-16T12:50:35Z

Looks good thanks for the addition, can you confirm all tests are passing with openvino-nightly (or another version enabling stateful models) before we can merge ?

I can confirm, I test this locally on my working machine

@eaidova Do the tests that fail with nightly here also pass for you locally with nightly/2023.3? https://github.com/huggingface/optimum-intel/actions/runs/7473712623/job/20526199806

FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_0_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_1_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_2_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int4_3_text_generation_with_past - AssertionError: 295 != 302
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_00_text_generation - AssertionError: 44 != 46
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_01_text_generation_with_past - AssertionError: 44 != 46
FAILED tests/openvino/test_exporters_cli.py::OVCLIExportTestCase::test_exporters_cli_int8_09_fill_mask - AssertionError: 68 != 70
FAILED tests/openvino/test_quantization.py::OVQuantizerTest::test_ovmodel_static_quantization_1 - AssertionError: 22 != 23
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_automodel_weight_compression_1 - AssertionError: 45 != 44
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_4bit_weight_compression_0 - AssertionError: 295 != 302
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_8bit_weight_compression_1 - AssertionError: 44 != 46
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_load_with_compressed_weights_00 - AssertionError: 44 != 46
FAILED tests/openvino/test_quantization.py::OVWeightCompressionTest::test_ovmodel_load_with_compressed_weights_01 - AssertionError: 68 != 70

@helena-intel I mean all relevant tests passed. I do not take it into account as it reproducible without stateful models and does not related to my changes. it is probably related to difference model graph representation for quantization

echarlaix · 2024-01-16T12:59:35Z

OK perfect, thanks a lot for checking @helena-intel @eaidova

slyalin added 17 commits December 5, 2023 09:32

Allow loading of stateful models (no patching yet)

d41110c

Stateful models support

bcb7cac

Fix forward for chatglm

d4c165b

Passing stateful as a dedicated parameter

a62dc6f

Fixed possibly misaligned types in ShapeOf Concat sub-expression

403adb5

Merge remote-tracking branch 'origin/main' into stateful

6097bfd

Fixed critical typo in infer_request invocation

c54a466

Merge remote-tracking branch 'origin/main' into stateful

6151bec

Apply bettertransfomer when model is converted in stateful mode

0e5aefc

Correct default value handling for stateful flag

5087f92

Apply bettertransformer under try-except to avoid crashes when model …

5ae6d5c

…is not supported

Added --stateful option in optimum-cli

ddb182e

Merge remote-tracking branch 'origin/main' into stateful

8e9a7e0

Raise if too old version of opevino is used ans stateful=True

a51ab27

Fix openvino version check to be compatible with openvino-nightly

6df798a

Merged from recent main branch

fa33784

Fix for bloom family

aa299e6

eaidova mentioned this pull request Dec 21, 2023

Stateful models by default #486

Closed

5 tasks

AlexKoff88 requested review from echarlaix and AlexKoff88 December 25, 2023 06:19

AlexKoff88 reviewed Dec 25, 2023

View reviewed changes

optimum/exporters/openvino/stateful.py Outdated Show resolved Hide resolved

AlexKoff88 reviewed Dec 25, 2023

View reviewed changes

optimum/exporters/openvino/better_transformer_patch.py Outdated Show resolved Hide resolved

AlexKoff88 reviewed Dec 25, 2023

View reviewed changes

tests/openvino/test_modeling.py Outdated Show resolved Hide resolved

eaidova force-pushed the ea/stateful branch from 7e1ce6f to b4a37fe Compare December 25, 2023 08:43

eaidova requested a review from AlexKoff88 December 26, 2023 13:24

slyalin added 3 commits December 28, 2023 19:13

Allow loading of stateful models (no patching yet)

70ff804

Stateful models support

bf45cc7

Fix forward for chatglm

fafc040

eaidova force-pushed the ea/stateful branch from 50a3f22 to 70d086a Compare January 8, 2024 13:40

eaidova added 4 commits January 8, 2024 18:56

fix test and add beam_idx attribute

6014294

apply review comments

045cc69

stateful by default fixes

2c3e934

less agressive stateful

27c0f0a

eaidova requested a review from slyalin January 10, 2024 11:54

eaidova added 2 commits January 10, 2024 17:40

ensure that task support stateful

18caf84

remove debug print

b059a5f

slyalin approved these changes Jan 12, 2024

View reviewed changes

slyalin reviewed Jan 12, 2024

View reviewed changes

optimum/exporters/openvino/convert.py Outdated Show resolved Hide resolved

slyalin reviewed Jan 12, 2024

View reviewed changes

optimum/commands/export/openvino.py Outdated Show resolved Hide resolved

Apply suggestions from code review

dd9f41d

Co-authored-by: Sergey Lyalin <[email protected]>

helena-intel approved these changes Jan 12, 2024

View reviewed changes

AlexKoff88 approved these changes Jan 15, 2024

View reviewed changes

eaidova and others added 2 commits January 15, 2024 11:22

Apply suggestions from code review

7006264

Co-authored-by: Helena Kloosterman <[email protected]>

update requirements and warning messages

2fb9032

echarlaix reviewed Jan 15, 2024

View reviewed changes

Apply suggestions from code review

b5e87b2

Co-authored-by: Ella Charlaix <[email protected]>

eaidova force-pushed the ea/stateful branch from 97005c4 to b5e87b2 Compare January 15, 2024 11:57

fix cli export

a18d57c

eaidova force-pushed the ea/stateful branch from 0716ed0 to a18d57c Compare January 15, 2024 14:00

echarlaix approved these changes Jan 16, 2024

View reviewed changes

optimum/exporters/openvino/__main__.py Outdated Show resolved Hide resolved

Update optimum/exporters/openvino/__main__.py

f79b4f1

Co-authored-by: Ella Charlaix <[email protected]>

echarlaix merged commit 7f236c2 into huggingface:main Jan 16, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support stateful model #493

support stateful model #493

eaidova commented Dec 21, 2023 •

edited

Loading

HuggingFaceDocBuilderDev commented Dec 21, 2023

eaidova commented Dec 21, 2023

slyalin commented Jan 12, 2024

helena-intel left a comment

helena-intel Jan 12, 2024

AlexKoff88 commented Jan 16, 2024

echarlaix left a comment

eaidova commented Jan 16, 2024

helena-intel commented Jan 16, 2024

eaidova commented Jan 16, 2024

echarlaix commented Jan 16, 2024

	"Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path."
	"Provide a model that can be converted to OpenVINO without fallback to ONNX conversion path."

support stateful model #493

support stateful model #493

Conversation

eaidova commented Dec 21, 2023 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Dec 21, 2023

eaidova commented Dec 21, 2023

slyalin commented Jan 12, 2024

helena-intel left a comment

Choose a reason for hiding this comment

helena-intel Jan 12, 2024

Choose a reason for hiding this comment

AlexKoff88 commented Jan 16, 2024

echarlaix left a comment

Choose a reason for hiding this comment

eaidova commented Jan 16, 2024

helena-intel commented Jan 16, 2024

eaidova commented Jan 16, 2024

echarlaix commented Jan 16, 2024

eaidova commented Dec 21, 2023 •

edited

Loading