Skip to content

Commit

Permalink
add comment
Browse files Browse the repository at this point in the history
  • Loading branch information
echarlaix committed Oct 4, 2023
1 parent 2550929 commit ebeb90c
Showing 1 changed file with 3 additions and 10 deletions.
13 changes: 3 additions & 10 deletions docs/source/inference.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -96,15 +96,15 @@ tokenizer.save_pretrained(save_directory)

### Weight only quantization

You can also apply INT8 quantization on your models weights when exporting your model by adding `--int8`:
You can also apply INT8 quantization on your models weights when exporting your model with the CLI:

```bash
optimum-cli export openvino --model gpt2 --int8 ov_model
```

This will results in the exported model linear and embedding layers to be quanrtized to INT8, while the activations will be kept in floating point precision.
This will results in the exported model linear and embedding layers to be quanrtized to INT8, the activations will be kept in floating point precision.

This can also be done when loading your model by setting `load_in_8bit=True`:
This can also be done when loading your model by setting the `load_in_8bit` argument when calling the `from_pretrained()` method.

```python
from optimum.intel import OVModelForCausalLM
Expand Down Expand Up @@ -360,13 +360,6 @@ image.save("fantasy_landscape.png")
| `image-to-image` | `OVStableDiffusionXLImg2ImgPipeline` |


Before using `OVtableDiffusionXLPipeline` make sure to have `diffusers` and `invisible_watermark` installed. You can install the libraries as follows:

```bash
pip install diffusers
pip install invisible-watermark>=0.2.0
```

#### Text-to-Image

Here is an example of how you can load a SDXL OpenVINO model from [stabilityai/stable-diffusion-xl-base-1.0](https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0) and run inference using OpenVINO Runtime:
Expand Down

0 comments on commit ebeb90c

Please sign in to comment.