Skip to content

Commit

Permalink
add int8
Browse files Browse the repository at this point in the history
  • Loading branch information
echarlaix committed Oct 4, 2023
1 parent 09c2e00 commit 0654305
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,13 @@ It is possible to export your model to the [OpenVINO](https://docs.openvino.ai/2
optimum-cli export openvino --model distilbert-base-uncased-finetuned-sst-2-english ov_distilbert
```

To apply int8 quantization on your model weights and keep the activations in floating point precision, you can add `--int8`:

```plain
optimum-cli export openvino --model distilbert-base-uncased-finetuned-sst-2-english --int8 ov_distilbert
```


#### Inference:

To load a model and run inference with OpenVINO Runtime, you can just replace your `AutoModelForXxx` class with the corresponding `OVModelForXxx` class.
Expand Down

0 comments on commit 0654305

Please sign in to comment.