core[patch], langchain[patch]: ByteStore (#14312)

langchain-ai · Dec 6, 2023 · 7bdfc43 · 7bdfc43
1 parent b9087e7
commit 7bdfc43
Show file tree

Hide file tree

Showing 19 changed files with 497 additions and 482 deletions.
diff --git a/docs/.local_build.sh b/docs/.local_build.sh
@@ -9,7 +9,7 @@ SCRIPT_DIR="$(cd "$(dirname "$0")"; pwd)"
 cd "${SCRIPT_DIR}"
 
 mkdir -p ../_dist
-rsync -ruv --exclude node_modules . ../_dist
+rsync -ruv --exclude node_modules --exclude api_reference --exclude .venv --exclude .docusaurus . ../_dist
 cd ../_dist
 poetry run python scripts/model_feat_table.py
 cp ../cookbook/README.md src/pages/cookbook.mdx

diff --git a/docs/docs/integrations/providers/redis.mdx b/docs/docs/integrations/providers/redis.mdx
@@ -17,6 +17,24 @@ Install the Python SDK:
 pip install redis
 ```
 
+To run Redis locally, you can use Docker:
+
+```bash
+docker run --name langchain-redis -d -p 6379:6379 redis redis-server --save 60 1 --loglevel warning
+```
+
+To stop the container:
+
+```bash
+docker stop langchain-redis
+```
+
+And to start it again:
+
+```bash
+docker start langchain-redis
+```
+
 ## Wrappers
 
 All wrappers need a redis url connection string to connect to the database support either a stand alone Redis server

diff --git a/docs/docs/integrations/stores/file_system.ipynb b/docs/docs/integrations/stores/file_system.ipynb
@@ -0,0 +1,100 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: Local Filesystem\n",
+    "sidebar_position: 3\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LocalFileStore\n",
+    "\n",
+    "The `LocalFileStore` is a persistent implementation of `ByteStore` that stores everything in a folder of your choosing."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[b'v1', b'v2']\n"
+     ]
+    }
+   ],
+   "source": [
+    "from pathlib import Path\n",
+    "\n",
+    "from langchain.storage import LocalFileStore\n",
+    "\n",
+    "root_path = Path.cwd() / \"data\"  # can also be a path set by a string\n",
+    "store = LocalFileStore(root_path)\n",
+    "\n",
+    "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+    "print(store.mget([\"k1\", \"k2\"]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's see which files exist in our `data` folder:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "k1 k2\n"
+     ]
+    }
+   ],
+   "source": [
+    "!ls {root_path}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/docs/integrations/stores/in_memory.ipynb b/docs/docs/integrations/stores/in_memory.ipynb
@@ -0,0 +1,73 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: In Memory\n",
+    "sidebar_position: 2\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# InMemoryByteStore\n",
+    "\n",
+    "The `InMemoryByteStore` is a non-persistent implementation of `ByteStore` that stores everything in a Python dictionary."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[b'v1', b'v2']\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.storage import InMemoryByteStore\n",
+    "\n",
+    "store = InMemoryByteStore()\n",
+    "\n",
+    "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+    "print(store.mget([\"k1\", \"k2\"]))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/docs/docs/integrations/stores/index.mdx b/docs/docs/integrations/stores/index.mdx
@@ -0,0 +1,29 @@
+---
+sidebar_position: 1
+sidebar_class_name: hidden
+---
+
+# Stores
+
+In many different applications, having some sort of key-value storage is helpful. 
+In this section, we will look at a few different ways to store key-value pairs
+using implementations of the `ByteStore` interface.
+
+## Features (natively supported)
+
+All `ByteStore`s support the following functions, which are used for modifying
+**m**ultiple key-value pairs at once:
+
+- `mget(key: Sequence[str]) -> List[Optional[bytes]]`: get the contents of multiple keys, returning `None` if the key does not exist
+- `mset(key_value_pairs: Sequence[Tuple[str, bytes]]) -> None`: set the contents of multiple keys
+- `mdelete(key: Sequence[str]) -> None`: delete multiple keys
+- `yield_keys(prefix: Optional[str] = None) -> Iterator[str]`: yield all keys in the store, optionally filtering by a prefix
+
+## How to pick one
+
+`ByteStore`s are designed to be interchangeable. By default, most dependent integrations
+use the `InMemoryByteStore`, which is a simple in-memory key-value store.
+
+However, if you start having other requirements, like massive scalability or persistence,
+you can swap out the `ByteStore` implementation with one of the other ones documented
+in this section.
diff --git a/docs/docs/integrations/stores/redis.ipynb b/docs/docs/integrations/stores/redis.ipynb
@@ -0,0 +1,83 @@
+{
+ "cells": [
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "---\n",
+    "sidebar_label: Redis\n",
+    "---"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# RedisStore\n",
+    "\n",
+    "The `RedisStore` is an implementation of `ByteStore` that stores everything in your Redis instance.\n",
+    "\n",
+    "To configure Redis, follow our [Redis guide](/docs/integrations/providers/redis)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install redis"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "[b'v1', b'v2']\n"
+     ]
+    }
+   ],
+   "source": [
+    "from langchain.storage import RedisStore\n",
+    "\n",
+    "store = RedisStore(redis_url=\"redis://localhost:6379\")\n",
+    "\n",
+    "store.mset([(\"k1\", b\"v1\"), (\"k2\", b\"v2\")])\n",
+    "print(store.mget([\"k1\", \"k2\"]))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.4"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}