Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Push the OptimumEmbedders to their performance limits #384

Closed
mathislucka opened this issue Feb 9, 2024 · 1 comment · Fixed by #496
Closed

Push the OptimumEmbedders to their performance limits #384

mathislucka opened this issue Feb 9, 2024 · 1 comment · Fixed by #496
Assignees
@shadeMe
Labels
P1

Comments

@mathislucka
Copy link
Member

Background
The OptimumEmbedders optimize inference speed for embedding inference. We have a basic version almost ready. In this task, we want to push these embedding components to their performance limits.

Ideas

  • optimize data loading
  • optimize batching
  • optimize TensorRT engine build
  • try FP16 inference
  • play with further optimizations (ORTOptimizer, ORTQuantizer)
@masci masci added the P1 label Feb 9, 2024
@shadeMe
Copy link
Contributor

shadeMe commented Feb 12, 2024

Blocked on #137

@shadeMe shadeMe linked a pull request Feb 28, 2024 that will close this issue
feat!: Add support for Optimum optimizers and quantizers #496
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shadeMe shadeMe

Labels
P1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat!: Add support for Optimum optimizers and quantizers deepset-ai/haystack-core-integrations
3 participants
@masci @shadeMe @mathislucka