Prompt Caching? #9444
Replies: 2 comments 8 replies
-
|
@brandonh-msft - thanks for brining this up. This looks like it is happening on the model end so you should be getting this already. Cosmos DB and Redis also supports Semantic Caching if you want to check that out also. You can also do Semantic Caching with filters also here - |
Beta Was this translation helpful? Give feedback.
-
|
A related question would be - is it possible for SK to report Cached tokens in the trace messages when hooked up with Open Telemetry. It would be nice to get that value along with the info generated:
Basically, everything OpenAI returns (including cached tokens, like below): |
Beta Was this translation helpful? Give feedback.
-
|
@TaoChenOSU - can you see if we can pull in the additional info from OpenAI. |
Beta Was this translation helpful? Give feedback.
-
|
@brandonh-msft - this will depend how many tokens you are using with your plugin description and the length of chat history. |
Beta Was this translation helpful? Give feedback.
-
|
i understand that, but it won't do anything at all if we aren't structuring the request to take advantage of it. this is what I opened this discussion to ask. From the responses thus far it seems like the straightforward answer is "no, we aren't" so I will probably transition this to an issue if/when have time to dissect the code & confirm. (unless one of the other participants here wants to do that 🙏🏻) |
Beta Was this translation helpful? Give feedback.
-
|
It looks like this behavior is controlled clear down in the OpenAI pkg for .NET and so it's on them to make sure this is being done. I will open discussion/issue accordingly. |
Beta Was this translation helpful? Give feedback.
-
|
Issue filed on OpenAI .NET SDK: openai/openai-dotnet#281 |
Beta Was this translation helpful? Give feedback.
-
Prompt caching for AOAI and OAI has been released, but has some requirements around how to make it most effective, specifically, the order in which things are serialized.
Is SK already serializing objects in this manner so token caching will get maximum hits? If not, is there work on the backlog to do it and when could we expect it (for each lang)?
Beta Was this translation helpful? Give feedback.
All reactions