Skip to content

Commit

Permalink
Add overflow_fix explanation to notebook (#273)
Browse files Browse the repository at this point in the history
  • Loading branch information
helena-intel authored Apr 18, 2023
1 parent 357aa81 commit 86ba447
Showing 1 changed file with 15 additions and 3 deletions.
18 changes: 15 additions & 3 deletions notebooks/openvino/question_answering_quantization.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,19 @@
"id": "eb0b0738-fdc9-4557-97bf-b4c6709280cc",
"metadata": {},
"source": [
"### Quantize the Model with Post Training Quantization"
"### Quantize the Model with Post Training Quantization\n",
"\n",
"**NOTE:** if you notice very low accuracy after post-training quantization, it is likely caused by an overflow issue which affects processors that do not contain VNNI (Vector Neural Network Instruction). NNCF has an `overflow_fix` option to address this. It will effectively use 7-bits for quantizing instead of 8-bits to prevent the overflow. To use this option, modify the code in the next cell to add an explicit quantization configuration, and set `overflow_fix` to `\"enable\"`:\n",
"\n",
"```\n",
"from optimum.intel.openvino import OVConfig\n",
"\n",
"ov_config = OVConfig()\n",
"ov_config.compression[\"overflow_fix\"] = \"enable\"\n",
"quantizer = OVQuantizer.from_pretrained(model, ov_config=ov_config)\n",
"```\n",
"\n",
"For more information, see [Lower Numerical Precision Deep Learning Inference and Training](https://www.intel.com/content/www/us/en/developer/articles/technical/lower-numerical-precision-deep-learning-inference-and-training.html)"
]
},
{
Expand Down Expand Up @@ -936,7 +948,7 @@
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -950,7 +962,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.16"
"version": "3.8.10"
}
},
"nbformat": 4,
Expand Down

0 comments on commit 86ba447

Please sign in to comment.