Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Core: Handle Failed Cache Reads For Non-Serializable Objects #27989

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
+22 −2
Draft

Core: Handle Failed Cache Reads For Non-Serializable Objects #27989

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
dda39ed
added check for cached result being of type ChatGeneration[]
FarhanChowdhury248 Nov 8, 2024
8094184
fixed linting issue
FarhanChowdhury248 Nov 8, 2024
54a9978
reduced long lines
FarhanChowdhury248 Nov 8, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions libs/core/langchain_core/language_models/chat_models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -814,7 +814,17 @@ def _generate_with_cache(
llm_string = self._get_llm_string(stop=stop, **kwargs)
prompt = dumps(messages)
cache_val = llm_cache.lookup(prompt, llm_string)
if isinstance(cache_val, list):
# When objects that are not Serializable are cached,
# they may not be succesfully serialized upon read.
# Currently, many caches failover to returning a list
# of Generation objects in this case, which is invalid
# for ChatResult, so we check and treat this as an
# uncached case.
if (
isinstance(cache_val, list)
and len(cache_val) > 0
and isinstance(cache_val[0], ChatGeneration)
):
return ChatResult(generations=cache_val)
elif self.cache is None:
pass
Expand Down Expand Up @@ -887,7 +897,17 @@ async def _agenerate_with_cache(
llm_string = self._get_llm_string(stop=stop, **kwargs)
prompt = dumps(messages)
cache_val = await llm_cache.alookup(prompt, llm_string)
if isinstance(cache_val, list):
# When objects that are not Serializable are cached,
# they may not be succesfully serialized upon read.
# Currently, many caches failover to returning a list
# of Generation objects in this case, which is invalid
# for ChatResult, so we check and treat this as an
# uncached case.
if (
isinstance(cache_val, list)
and len(cache_val) > 0
and isinstance(cache_val[0], ChatGeneration)
):
return ChatResult(generations=cache_val)
elif self.cache is None:
pass
Expand Down
Loading