Skip to content

Commit

Permalink
docs[minor]: Update more vector store docs (#6444)
Browse files Browse the repository at this point in the history 
* Update Pinecone docs

* Update Qdrant docs

* Update vector stores

* Fix
  • Loading branch information
@jacoblee93
jacoblee93 authored Aug 7, 2024
1 parent 31bfdc1 commit 46f83ed
Show file tree
Hide file tree
Showing 15 changed files with 2,279 additions and 413 deletions.
2 changes: 1 addition & 1 deletion docs/core_docs/docs/integrations/tools/tavily_search.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@
"\n",
"| Class | Package | [PY support](https://python.langchain.com/v0.2/docs/integrations/tools/tavily_search/) | Package latest |\n",
"| :--- | :--- | :---: | :---: | :---: |\n",
"| [TavilySearchResults](https://api.js.langchain.com/classes/langchain_community_tools_tavily_search.TavilySearchResults.html) | [`@langchain/community`](https://www.npmjs.com/package/@langchain/community) | ✅ | ![NPM - Version](https://img.shields.io/npm/v/__package_name__?style=flat-square&label=%20&) |\n",
"| [TavilySearchResults](https://api.js.langchain.com/classes/langchain_community_tools_tavily_search.TavilySearchResults.html) | [`@langchain/community`](https://www.npmjs.com/package/@langchain/community) | ✅ | ![NPM - Version](https://img.shields.io/npm/v/@langchain/community?style=flat-square&label=%20&) |\n",
"\n",
"## Setup\n",
"\n",
Expand Down
2 changes: 2 additions & 0 deletions docs/core_docs/docs/integrations/vectorstores/mongodb_atlas.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,8 @@
"id": "dcf1b905",
"metadata": {},
"source": [
"**Note:** After adding documents, there is a slight delay before they become queryable.\n",
"\n",
"Adding a document with the same `id` as an existing document will update the existing one.\n",
"\n",
"### Delete items from vector store"
Expand Down
366 changes: 366 additions & 0 deletions docs/core_docs/docs/integrations/vectorstores/pinecone.ipynb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,366 @@
{
"cells": [
{
"cell_type": "raw",
"id": "1957f5cb",
"metadata": {
"vscode": {
"languageId": "raw"
}
},
"source": [
"---\n",
"sidebar_label: Pinecone\n",
"---"
]
},
{
"cell_type": "markdown",
"id": "ef1f0986",
"metadata": {},
"source": [
"# PineconeStore\n",
"\n",
"[Pinecone](https://www.pinecone.io/) is a vector database that helps power AI for some of the world’s best companies.\n",
"\n",
"This guide provides a quick overview for getting started with Pinecone [vector stores](/docs/concepts/#vectorstores). For detailed documentation of all `PineconeStore` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_pinecone.PineconeStore.html)."
]
},
{
"cell_type": "markdown",
"id": "c824838d",
"metadata": {},
"source": [
"## Overview\n",
"\n",
"### Integration details\n",
"\n",
"| Class | Package | [PY support](https://python.langchain.com/v0.2/docs/integrations/vectorstores/pinecone/) | Package latest |\n",
"| :--- | :--- | :---: | :---: |\n",
"| [`PineconeStore`](https://api.js.langchain.com/classes/langchain_pinecone.PineconeStore.html) | [`@langchain/pinecone`](https://npmjs.com/@langchain/pinecone) | ✅ | ![NPM - Version](https://img.shields.io/npm/v/@langchain/pinecone?style=flat-square&label=%20&) |"
]
},
{
"cell_type": "markdown",
"id": "36fdc060",
"metadata": {},
"source": [
"## Setup\n",
"\n",
"To use Pinecone vector stores, you'll need to create a Pinecone account, initialize an index, and install the `@langchain/pinecone` integration package. You'll also want to install the [official Pinecone SDK](https://www.npmjs.com/package/@pinecone-database/pinecone) to initialize a client to pass into the `PineconeStore` instance.\n",
"\n",
"This guide will also use [OpenAI embeddings](/docs/integrations/text_embedding/openai), which require you to install the `@langchain/openai` integration package. You can also use [other supported embeddings models](/docs/integrations/text_embedding) if you wish.\n",
"\n",
"```{=mdx}\n",
"import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n",
"import Npm2Yarn from \"@theme/Npm2Yarn\";\n",
"\n",
"<IntegrationInstallTooltip></IntegrationInstallTooltip>\n",
"\n",
"<Npm2Yarn>\n",
" @langchain/pinecone @pinecone-database/pinecone @langchain/openai\n",
"</Npm2Yarn>\n",
"```\n",
"\n",
"### Credentials\n",
"\n",
"Sign up for a [Pinecone](https://www.pinecone.io/) account and create an index. Make sure the dimensions match those of the embeddings you want to use (the default is 1536 for OpenAI's `text-embedding-3-small`). Once you've done this set the `PINECONE_INDEX`, `PINECONE_API_KEY`, and (optionally) `PINECONE_ENVIRONMENT` environment variables:\n",
"\n",
"```typescript\n",
"process.env.PINECONE_API_KEY = \"your-pinecone-api-key\";\n",
"process.env.PINECONE_INDEX = \"your-pinecone-index\";\n",
"\n",
"// Optional\n",
"process.env.PINECONE_ENVIRONMENT = \"your-pinecone-environment\";\n",
"```\n",
"\n",
"If you are using OpenAI embeddings for this guide, you'll need to set your OpenAI key as well:\n",
"\n",
"```typescript\n",
"process.env.OPENAI_API_KEY = \"YOUR_API_KEY\";\n",
"```\n",
"\n",
"If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n",
"\n",
"```typescript\n",
"// process.env.LANGCHAIN_TRACING_V2=\"true\"\n",
"// process.env.LANGCHAIN_API_KEY=\"your-api-key\"\n",
"```"
]
},
{
"cell_type": "markdown",
"id": "93df377e",
"metadata": {},
"source": [
"## Instantiation"
]
},
{
"cell_type": "code",
"execution_count": 1,
"id": "dc37144c-208d-4ab3-9f3a-0407a69fe052",
"metadata": {
"tags": []
},
"outputs": [],
"source": [
"import { PineconeStore } from \"@langchain/pinecone\";\n",
"import { OpenAIEmbeddings } from \"@langchain/openai\";\n",
"\n",
"import { Pinecone as PineconeClient } from \"@pinecone-database/pinecone\";\n",
"\n",
"const embeddings = new OpenAIEmbeddings({\n",
" model: \"text-embedding-3-small\",\n",
"});\n",
"\n",
"const pinecone = new PineconeClient();\n",
"// Will automatically read the PINECONE_API_KEY and PINECONE_ENVIRONMENT env vars\n",
"const pineconeIndex = pinecone.Index(process.env.PINECONE_INDEX!);\n",
"\n",
"const vectorStore = await PineconeStore.fromExistingIndex(\n",
" embeddings,\n",
" {\n",
" pineconeIndex,\n",
" // Maximum number of batch requests to allow at once. Each batch is 1000 vectors.\n",
" maxConcurrency: 5,\n",
" // You can pass a namespace here too\n",
" // namespace: \"foo\",\n",
" }\n",
");"
]
},
{
"cell_type": "markdown",
"id": "ac6071d4",
"metadata": {},
"source": [
"## Manage vector store\n",
"\n",
"### Add items to vector store"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "17f5efc0",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ '1', '2', '3', '4' ]\n"
]
}
],
"source": [
"import type { Document } from \"@langchain/core/documents\";\n",
"\n",
"const document1: Document = {\n",
" pageContent: \"The powerhouse of the cell is the mitochondria\",\n",
" metadata: { source: \"https://example.com\" }\n",
"};\n",
"\n",
"const document2: Document = {\n",
" pageContent: \"Buildings are made out of brick\",\n",
" metadata: { source: \"https://example.com\" }\n",
"};\n",
"\n",
"const document3: Document = {\n",
" pageContent: \"Mitochondria are made out of lipids\",\n",
" metadata: { source: \"https://example.com\" }\n",
"};\n",
"\n",
"const document4: Document = {\n",
" pageContent: \"The 2024 Olympics are in Paris\",\n",
" metadata: { source: \"https://example.com\" }\n",
"}\n",
"\n",
"const documents = [document1, document2, document3, document4];\n",
"\n",
"await vectorStore.addDocuments(documents, { ids: [\"1\", \"2\", \"3\", \"4\"] });"
]
},
{
"cell_type": "markdown",
"id": "dcf1b905",
"metadata": {},
"source": [
"**Note:** After adding documents, there is a slight delay before they become queryable.\n",
"\n",
"### Delete items from vector store"
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "ef61e188",
"metadata": {},
"outputs": [],
"source": [
"await vectorStore.delete({ ids: [\"4\"] });"
]
},
{
"cell_type": "markdown",
"id": "c3620501",
"metadata": {},
"source": [
"## Query vector store\n",
"\n",
"Once your vector store has been created and the relevant documents have been added you will most likely wish to query it during the running of your chain or agent. \n",
"\n",
"### Query directly\n",
"\n",
"Performing a simple similarity search can be done as follows:"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "aa0a16fa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"* The powerhouse of the cell is the mitochondria [{\"source\":\"https://example.com\"}]\n",
"* Mitochondria are made out of lipids [{\"source\":\"https://example.com\"}]\n"
]
}
],
"source": [
"// Optional filter\n",
"const filter = { source: \"https://example.com\" };\n",
"\n",
"const similaritySearchResults = await vectorStore.similaritySearch(\"biology\", 2, filter);\n",
"\n",
"for (const doc of similaritySearchResults) {\n",
" console.log(`* ${doc.pageContent} [${JSON.stringify(doc.metadata, null)}]`);\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "3ed9d733",
"metadata": {},
"source": [
"If you want to execute a similarity search and receive the corresponding scores you can run:"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "5efd2eaa",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"* [SIM=0.165] The powerhouse of the cell is the mitochondria [{\"source\":\"https://example.com\"}]\n",
"* [SIM=0.148] Mitochondria are made out of lipids [{\"source\":\"https://example.com\"}]\n"
]
}
],
"source": [
"const similaritySearchWithScoreResults = await vectorStore.similaritySearchWithScore(\"biology\", 2, filter)\n",
"\n",
"for (const [doc, score] of similaritySearchWithScoreResults) {\n",
" console.log(`* [SIM=${score.toFixed(3)}] ${doc.pageContent} [${JSON.stringify(doc.metadata)}]`);\n",
"}"
]
},
{
"cell_type": "markdown",
"id": "0c235cdc",
"metadata": {},
"source": [
"### Query by turning into retriever\n",
"\n",
"You can also transform the vector store into a [retriever](/docs/concepts/#retrievers) for easier usage in your chains. "
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "f3460093",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[\n",
" Document {\n",
" pageContent: 'The powerhouse of the cell is the mitochondria',\n",
" metadata: { source: 'https://example.com' },\n",
" id: undefined\n",
" },\n",
" Document {\n",
" pageContent: 'Mitochondria are made out of lipids',\n",
" metadata: { source: 'https://example.com' },\n",
" id: undefined\n",
" }\n",
"]\n"
]
}
],
"source": [
"const retriever = vectorStore.asRetriever({\n",
" // Optional filter\n",
" filter: filter,\n",
" k: 2,\n",
"});\n",
"\n",
"await retriever.invoke(\"biology\");"
]
},
{
"cell_type": "markdown",
"id": "e2e0a211",
"metadata": {},
"source": [
"### Usage for retrieval-augmented generation\n",
"\n",
"For guides on how to use this vector store for retrieval-augmented generation (RAG), see the following sections:\n",
"\n",
"- [Tutorials: working with external knowledge](/docs/tutorials/#working-with-external-knowledge).\n",
"- [How-to: Question and answer with RAG](/docs/how_to/#qa-with-rag)\n",
"- [Retrieval conceptual docs](/docs/concepts#retrieval)"
]
},
{
"cell_type": "markdown",
"id": "8a27244f",
"metadata": {},
"source": [
"## API reference\n",
"\n",
"For detailed documentation of all `PineconeStore` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_pinecone.PineconeStore.html)."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "TypeScript",
"language": "typescript",
"name": "tslab"
},
"language_info": {
"codemirror_mode": {
"mode": "typescript",
"name": "javascript",
"typescript": true
},
"file_extension": ".ts",
"mimetype": "text/typescript",
"name": "typescript",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
Loading

0 comments on commit 46f83ed

Please sign in to comment.