Add Support for Memcached as a LLM Model Cache

[prokopchukdim]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

We would like to add support for Memcached as a usable LLM model cache. There are two main pure-Python memcached client implementations in Python: pymemcache and python-memcached.

We would primarily like to add support for pymemcache given that it is the most actively maintained, but it may be possible to support both clients under one newly added cache class since both are used.

Motivation

Many of the model caches supported natively are full on DBs. While Redis is supported as an option for distributed in-memory storage, many teams and companies rely on Memcached as a distributed in-memory cache. By adding Memcached support, we hope to make the model caching feature more useful to more teams using Langchain.

Example Usage

from langchain.globals import set_llm_cache
from langchain_openai import OpenAI

from langchain_community.cache import MemcachedCache
from pymemcache.client.base import Client

llm = OpenAI(model="gpt-3.5-turbo-instruct", n=2, best_of=2)
set_llm_cache(MemcachedCache(Client('localhost')))

# The first time, it is not yet in cache, so it should take longer
llm.invoke("Which city is the most crowded city in the USA?")

# The second time it is, so it goes faster
llm.invoke("Which city is the most crowded city in the USA?")

Proposal (If applicable)

We intend to add a new MemcachedCache implementation in libs/community/langchain_community/cache.py to support the pymemcache client.

If there is interest in also supporting the python-memcached client, or others, we can explore creating a unified implementation class since all clients should generally adhere to the memcached text protocol.

We intend to submit a pull request some time in October, and no later than mid-November.

[prokopchukdim]

If we were to add support for more than one client, would it be best to have one class per client? Or, since these would all be Memcached implementations, could we have one unified class which takes as part of its __init__ method some enum identifier for the client library, along with the client itself, and implements the relevant methods in a strategy-esque pattern?

The second option would provide a cleaner interface for end-users, but may be somewhat overengineering the implementation.