Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving inference time #109

Open
alokpadhi opened this issue Feb 15, 2024 · 2 comments
Open

Improving inference time #109

alokpadhi opened this issue Feb 15, 2024 · 2 comments

Comments

@alokpadhi
Copy link

I am using the Instructor Base model and did the quantization on top of it to improve the inference time. But even after doing the quantization the inference time is between 6-7 secs. Whereas based on my required I need to make it under 1 sec. Are there any other ways to improve the inference time of the model?

Server configuration:

  • Memory: 8 GB
  • CPUs: 4 cores
@EricPaul03
Copy link

Hello, I'm also seeking this kind of speed improvement. Do you have any good methods to share in the end?

@BBC-Esq
Copy link
Contributor

BBC-Esq commented Aug 12, 2024

You can use something like this:

model.client[0].auto_model = model.client[0].auto_model.to(torch_dtype)

However, you'll need to import the following:

from torch.cuda.amp import autocast, GradScaler

You can use it after instantiating the model. Unfortunately, I was unable to find a way to use the stereotypical torch.dtype approach.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants