Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support prompt caching in Anthropic generators #1004

Closed
julian-risch opened this issue Aug 19, 2024 · 3 comments · Fixed by #1006
Closed

Support prompt caching in Anthropic generators #1004

julian-risch opened this issue Aug 19, 2024 · 3 comments · Fixed by #1006
Assignees
Labels
feature request Ideas to improve an integration integration:anthropic P2

Comments

@julian-risch
Copy link
Member

Is your feature request related to a problem? Please describe.
Anthropic, in particular the models Claude 3.5 Sonnet and Claude 3 Haiku support prompt caching or context caching. Support for Claude 3 Opus is coming soon. We should enable users to use that feature through Haystack to reduce costs and latency.
https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching
https://www.anthropic.com/news/prompt-caching

Describe the solution you'd like
We need to implement support for Anthropic's cache_control parameter in messages.

@julian-risch
Copy link
Member Author

related issue #1003

@vblagoje
Copy link
Member

vblagoje commented Sep 2, 2024

To have a full benefit of prompt caching in Haystack pipelines we need to resolve #1011

@vblagoje
Copy link
Member

vblagoje commented Sep 20, 2024

Support for prompt caching added in https://pypi.org/project/anthropic-haystack/ 1.1.0 release. See our example on how to leverage prompt caching in a "chat with a long research paper" use case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Ideas to improve an integration integration:anthropic P2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants