Can we have LLMs loaded on multiple GPUs via the HuggingFacePipeline.from_model_id function?

[kiran-paul98]

I am trying to load mosaicml's mpt 7b by using the HuggingFacePipeline.from_model_id function using the device parameter. But I am not able to load the entire model into one GPU due to Out of Memory error. I have 2 GPUs. Is it possible to load the model into multiple GPUs? I know that currently device parameter takes only integer arguments. Is there a workaround so that we can pass multiple GPU devices to the function, so that models can be loaded?

[j-o-h-n-w-e-i-s]

Is there anything to be gained in ingest, embeddings or chatting from just on GPU?

Are the instructions to get one GPU utilized pretty simple? E.G.

1. install GPU Drivers
2. install CUDA tookit
3. Instal cuDNN
4. Install GPU-enabled deep learning frameworks, tensorflow, torch torchvision torchaudio
5. configure privateGPT to utilize GPU
6. Verify GPU usage

Am I missing anything?