Added quantization for OUTETTS #2662
Conversation
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@KodiaqQ , what results do you get with the quantized model vs original on your machine? |
FP model generate time: 3.926095366012305 upd. Recalculated times with ignored scope. |
| "hf_model = OVHFModel(model_dir, device.value).model\n", | ||
| "dataset = nncf.Dataset(libritts, partial(transform_fn, interface=interface))\n", | ||
| "\n", | ||
| "quantized_model = nncf.quantize(\n", |
There was a problem hiding this comment.
I would suggest to use INT4 weight compression with dynamic quantization (A8W4). @KodiaqQ claim that the performance of such model is equal to the performance of the quantized model, but compression reate is higher for A8W4 model.
cc' @MaximProshin
There was a problem hiding this comment.
Please share numbers for both cases. If int4 is better, I'm ok to use that method.
There was a problem hiding this comment.
We discussed it offline and agreed to keep int8. At the same time the issue with SDPA will be analyzed in #16177. If resolved, we will update the notebook afterwards.
|
@l-bat, can you review, please? Thanks. |
| @@ -22,6 +22,11 @@ | |||
| "- [Run model inference](#Run-model-inference)\n", | |||
There was a problem hiding this comment.
Line #36. "__module.model.layers.*.self_attn/aten::scaled_dot_product_attention/ScaledDotProductAttention"
Could you please provide a reason in the description for why we should add this pattern to IgnoredScope?
Reply via ReviewNB
| @@ -22,6 +22,11 @@ | |||
| "- [Run model inference](#Run-model-inference)\n", | |||
There was a problem hiding this comment.
Line #3. demo = make_demo(interface)
I think it would be nice to use the quantized model in the interactive demo. Could you please add an option for the user to choose between the original and optimized models?
Reply via ReviewNB
| @@ -22,6 +22,11 @@ | |||
| "- [Run model inference](#Run-model-inference)\n", | |||
There was a problem hiding this comment.
Line #4. r = requests.get(
please add check that "skip_kernel_extension.py" exists to allow rerun notebook without internet connection
Reply via ReviewNB
|
@nikita-malininn could you please also fix formatting for updated notebook? P.S. after fixing code formatting and file downloading, I'm ready to merge your PR |
nncf.quantizeto the notebookmixedpreset andtransformermodel type due to model architecture (LLM)ignored scopedue to OpenVINO issues with the optimized SDPA inferencegeneratepipeline is possible because of the model architectureTicket: 157133