Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support stateful model #493

Merged
merged 84 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
d41110c
Allow loading of stateful models (no patching yet)
slyalin Dec 5, 2023
bcb7cac
Stateful models support
slyalin Dec 7, 2023
d4c165b
Fix forward for chatglm
slyalin Dec 7, 2023
a62dc6f
Passing stateful as a dedicated parameter
slyalin Dec 8, 2023
403adb5
Fixed possibly misaligned types in ShapeOf Concat sub-expression
slyalin Dec 8, 2023
6097bfd
Merge remote-tracking branch 'origin/main' into stateful
slyalin Dec 12, 2023
c54a466
Fixed critical typo in infer_request invocation
slyalin Dec 12, 2023
6151bec
Merge remote-tracking branch 'origin/main' into stateful
slyalin Dec 13, 2023
0e5aefc
Apply bettertransfomer when model is converted in stateful mode
slyalin Dec 13, 2023
5087f92
Correct default value handling for stateful flag
slyalin Dec 13, 2023
5ae6d5c
Apply bettertransformer under try-except to avoid crashes when model …
slyalin Dec 13, 2023
ddb182e
Added --stateful option in optimum-cli
slyalin Dec 13, 2023
8e9a7e0
Merge remote-tracking branch 'origin/main' into stateful
slyalin Dec 14, 2023
a51ab27
Raise if too old version of opevino is used ans stateful=True
slyalin Dec 14, 2023
6df798a
Fix openvino version check to be compatible with openvino-nightly
slyalin Dec 14, 2023
fa33784
Merged from recent main branch
slyalin Dec 19, 2023
aa299e6
Fix for bloom family
slyalin Dec 19, 2023
70ff804
Allow loading of stateful models (no patching yet)
slyalin Dec 5, 2023
bf45cc7
Stateful models support
slyalin Dec 7, 2023
fafc040
Fix forward for chatglm
slyalin Dec 7, 2023
e4585fe
Passing stateful as a dedicated parameter
slyalin Dec 8, 2023
a5c48c4
Fixed possibly misaligned types in ShapeOf Concat sub-expression
slyalin Dec 8, 2023
3ce4fe9
Fixed critical typo in infer_request invocation
slyalin Dec 12, 2023
00bd0b3
Apply bettertransfomer when model is converted in stateful mode
slyalin Dec 13, 2023
489799a
Correct default value handling for stateful flag
slyalin Dec 13, 2023
312f24f
Apply bettertransformer under try-except to avoid crashes when model …
slyalin Dec 13, 2023
4bca339
Added --stateful option in optimum-cli
slyalin Dec 13, 2023
a5f8558
Raise if too old version of opevino is used ans stateful=True
slyalin Dec 14, 2023
2763639
Fix openvino version check to be compatible with openvino-nightly
slyalin Dec 14, 2023
f5da152
Fix for bloom family
slyalin Dec 19, 2023
da62d99
Fix general code style and appliy renaming suggestions
eaidova Dec 21, 2023
33f86dc
fix version checking if openvino not in site-packages
eaidova Dec 21, 2023
7d15415
use reset_stateif available
eaidova Dec 21, 2023
0dfbb63
remove input patch in bettertransformer apply
eaidova Dec 21, 2023
09bbe7b
add tests
eaidova Dec 21, 2023
e70a50d
add type hints and update doc strings
eaidova Dec 21, 2023
a57c86a
added more tests
eaidova Dec 25, 2023
67deae2
Merge remote-tracking branch 'origin/main' into ea/stateful
slyalin Jan 4, 2024
89ba0cb
Fixed outdated signature of InferRequest wrapper to fix one of the qu…
slyalin Jan 4, 2024
a85960a
Merge remote-tracking branch 'slyalin/stateful' into stateful
slyalin Jan 4, 2024
dcfcc2a
Merge remote-tracking branch 'origin/main' into stateful
slyalin Jan 5, 2024
df0e727
Switch to stateful model by default
slyalin Jan 5, 2024
9992419
Merge remote-tracking branch 'origin/main' into stateful
slyalin Jan 5, 2024
1c815a0
Allow loading of stateful models (no patching yet)
slyalin Dec 5, 2023
132ae62
Stateful models support
slyalin Dec 7, 2023
fa02a9e
Fix forward for chatglm
slyalin Dec 7, 2023
9094279
Passing stateful as a dedicated parameter
slyalin Dec 8, 2023
14aeb55
Fixed possibly misaligned types in ShapeOf Concat sub-expression
slyalin Dec 8, 2023
6594e70
Apply bettertransfomer when model is converted in stateful mode
slyalin Dec 13, 2023
55c70fe
Correct default value handling for stateful flag
slyalin Dec 13, 2023
a1b41ac
Apply bettertransformer under try-except to avoid crashes when model …
slyalin Dec 13, 2023
a7d91b7
Added --stateful option in optimum-cli
slyalin Dec 13, 2023
2aeaf68
Raise if too old version of opevino is used ans stateful=True
slyalin Dec 14, 2023
72e9149
Fix openvino version check to be compatible with openvino-nightly
slyalin Dec 14, 2023
4488836
Fix for bloom family
slyalin Dec 19, 2023
843d364
Fix general code style and appliy renaming suggestions
eaidova Dec 21, 2023
361e06e
fix version checking if openvino not in site-packages
eaidova Dec 21, 2023
0b8ba34
use reset_stateif available
eaidova Dec 21, 2023
6b88888
remove input patch in bettertransformer apply
eaidova Dec 21, 2023
042944c
add tests
eaidova Dec 21, 2023
662f615
add type hints and update doc strings
eaidova Dec 21, 2023
c75e40b
added more tests
eaidova Dec 25, 2023
8531d3c
Fixed outdated signature of InferRequest wrapper to fix one of the qu…
slyalin Jan 4, 2024
b9d42a7
Stateful models support
slyalin Dec 7, 2023
841c8ec
Fix forward for chatglm
slyalin Dec 7, 2023
94ee68d
Passing stateful as a dedicated parameter
slyalin Dec 8, 2023
7ce3eb4
Apply bettertransfomer when model is converted in stateful mode
slyalin Dec 13, 2023
7ead0e8
Raise if too old version of opevino is used ans stateful=True
slyalin Dec 14, 2023
49a7f8f
Fix openvino version check to be compatible with openvino-nightly
slyalin Dec 14, 2023
3bf4272
Fix for bloom family
slyalin Dec 19, 2023
e2194b3
fix conflict
eaidova Jan 8, 2024
70d086a
Merge remote-tracking branch 'sl/stateful' into ea/stateful
eaidova Jan 8, 2024
6014294
fix test and add beam_idx attribute
eaidova Jan 8, 2024
045cc69
apply review comments
eaidova Jan 8, 2024
2c3e934
stateful by default fixes
eaidova Jan 10, 2024
27c0f0a
less agressive stateful
eaidova Jan 10, 2024
18caf84
ensure that task support stateful
eaidova Jan 10, 2024
b059a5f
remove debug print
eaidova Jan 12, 2024
dd9f41d
Apply suggestions from code review
eaidova Jan 12, 2024
7006264
Apply suggestions from code review
eaidova Jan 15, 2024
2fb9032
update requirements and warning messages
eaidova Jan 15, 2024
b5e87b2
Apply suggestions from code review
eaidova Jan 15, 2024
a18d57c
fix cli export
eaidova Jan 15, 2024
f79b4f1
Update optimum/exporters/openvino/__main__.py
eaidova Jan 16, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Apply suggestions from code review
Co-authored-by: Helena Kloosterman <helena.kloosterman@intel.com>
  • Loading branch information
eaidova and helena-intel authored Jan 15, 2024
commit 7006264288af13b4d80d442c7ac221d4ad2571bf
2 changes: 1 addition & 1 deletion optimum/commands/export/openvino.py
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ def parse_args_openvino(parser: "ArgumentParser"):
"Disable stateful converted models, stateless models will be generated instead. Stateful models are produced by default when this key is not used. "
"In stateful models all kv-cache inputs and outputs are hidden in the model and are not exposed as model inputs and outputs. "
"If --no-stateful option is used, it may result in sub-optimal inference performance. "
"Use it when you intentionally want to use a stateless model, for example, to be compatible with exiting "
"Use it when you intentionally want to use a stateless model, for example, to be compatible with existing "
"OpenVINO native inference code that expects kv-cache inputs and outputs in the model.",
),
)
Expand Down
2 changes: 1 addition & 1 deletion optimum/exporters/openvino/model_patcher.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ def patch_model_with_bettertransformer(model):
log.warn(
"integration Scaled Dot Product Attention optimization supported only with torch > 2.0."
"Usage model with stateful=True may be non-effective if model does not contain torch.functional.scaled_dot_product_attention"
"It is recommended to upgrade PyTorch version for using stateful model or use stateful=Flase"
"It is recommended to upgrade PyTorch version for using stateful model or use stateful=False"
)
# model already has required SDPA implementation
if getattr(model, "_supports_sdpa", False) and getattr(model.config, "_attn_implementation", "eager") == "sdpa":
Expand Down
2 changes: 1 addition & 1 deletion optimum/exporters/openvino/stateful.py
Original file line number Diff line number Diff line change
Expand Up @@ -206,7 +206,7 @@ def ensure_export_task_support_stateful(task: str):

def patch_stateful(config: PretrainedConfig, ov_model: ov.Model):
"""
Apply stateful transformation to model to hiding key values inputs inside model.
Apply stateful transformation to model to hide key values inputs inside model.
Select transformation parameters based on model architecture

Parameters:
Expand Down
Loading