From 653586159ecacfa102642db03866e38f40a9e598 Mon Sep 17 00:00:00 2001 From: Jacob Lee Date: Fri, 7 Jun 2024 13:23:58 -0700 Subject: [PATCH] docs[patch]: Add crosslinks, add LangGraph and LangSmith sections (#5702) * Add crosslinks, add LangGraph and LangSmith sections * Fix anchors --- docs/core_docs/docs/concepts.mdx | 57 ++++++++++++++++------ docs/core_docs/docs/how_to/index.mdx | 46 ++++++++++++----- docs/core_docs/docs/tutorials/index.mdx | 16 ++++++ docs/core_docs/docs/tutorials/pdf_qa.ipynb | 2 +- 4 files changed, 95 insertions(+), 26 deletions(-) diff --git a/docs/core_docs/docs/concepts.mdx b/docs/core_docs/docs/concepts.mdx index 91f267823241..513c840e5dbf 100644 --- a/docs/core_docs/docs/concepts.mdx +++ b/docs/core_docs/docs/concepts.mdx @@ -134,19 +134,6 @@ The **input type** and **output type** varies by component: LangChain provides standard, extendable interfaces and external integrations for various components useful for building with LLMs. Some components LangChain implements, some components we rely on third-party integrations for, and others are a mix. -### LLMs - - - -Language models that takes a string as input and returns a string. -These are traditionally older models (newer models generally are `ChatModels`, see below). - -Although the underlying models are string in, string out, the LangChain wrappers also allow these models to take messages as input. -This makes them interchangeable with ChatModels. -When messages are passed in as input, they will be formatted into a string under the hood before being passed to the underlying model. - -LangChain does not provide any LLMs, rather we rely on third party integrations. - ### Chat models @@ -167,6 +154,23 @@ We have some standardized parameters when constructing ChatModels: ChatModels also accept other parameters that are specific to that integration. +For specifics on how to use chat models, see the [relevant how-to guides here](/docs/how_to/#chat-models). + +### LLMs + + + +Language models that takes a string as input and returns a string. +These are traditionally older models (newer models generally are `ChatModels`, see below). + +Although the underlying models are string in, string out, the LangChain wrappers also allow these models to take messages as input. +This makes them interchangeable with ChatModels. +When messages are passed in as input, they will be formatted into a string under the hood before being passed to the underlying model. + +LangChain does not provide any LLMs, rather we rely on third party integrations. + +For specifics on how to use LLMs, see the [relevant how-to guides here](/docs/how_to/#llms). + ### Function/Tool Calling :::info @@ -267,7 +271,7 @@ Prompt Templates take as input an object, where each key represents a variable i Prompt Templates output a PromptValue. This PromptValue can be passed to an LLM or a ChatModel, and can also be cast to a string or an array of messages. The reason this PromptValue exists is to make it easy to switch between strings and messages. -There are a few different types of prompt templates +There are a few different types of prompt templates: #### String PromptTemplates @@ -341,6 +345,8 @@ const promptTemplate = ChatPromptTemplate.fromMessages([ ]); ``` +For specifics on how to use prompt templates, see the [relevant how-to guides here](/docs/how_to/#prompt-templates). + ### Example Selectors One common prompting technique for achieving better performance is to include examples as part of the prompt. @@ -348,6 +354,8 @@ This gives the language model concrete examples of how it should behave. Sometimes these examples are hardcoded into the prompt, but for more advanced situations it may be nice to dynamically select them. Example Selectors are classes responsible for selecting and then formatting examples into prompts. +For specifics on how to use example selectors, see the [relevant how-to guides here](/docs/how_to/#example-selectors). + ### Output parsers @@ -398,6 +406,8 @@ LangChain has many different types of output parsers. This is a list of output p | [Datetime](https://v02.api.js.langchain.com/classes/langchain_output_parsers.DatetimeOutputParser.html) | | `string` | `Promise` | Parses response into a `Date`. | | [Regex](https://v02.api.js.langchain.com/classes/langchain_output_parsers.RegexParser.html) | | `string` | `Promise>` | Parses the given text using the regex pattern and returns a object with the parsed output. | +For specifics on how to use output parsers, see the [relevant how-to guides here](/docs/how_to/#output-parsers). + ### Chat History Most LLM applications have a conversational interface. @@ -435,6 +445,8 @@ const loader = new CSVLoader(); const docs = await loader.load(); ``` +For specifics on how to use document loaders, see the [relevant how-to guides here](/docs/how_to/#document-loaders). + ### Text splitters Once you've loaded documents, you'll often want to transform them to better suit your application. The simplest example is you may want to split a long document into smaller chunks that can fit into your model's context window. LangChain has a number of built-in document transformers that make it easy to split, combine, filter, and otherwise manipulate documents. @@ -452,6 +464,8 @@ That means there are two different axes along which you can customize your text 1. How the text is split 2. How the chunk size is measured +For specifics on how to use text splitters, see the [relevant how-to guides here](/docs/how_to/#text-splitters). + ### Embedding models @@ -462,6 +476,8 @@ Embeddings create a vector representation of a piece of text. This is useful bec The base Embeddings class in LangChain provides two methods: one for embedding documents and one for embedding a query. The former takes as input multiple texts, while the latter takes a single text. The reason for having these as two separate methods is that some embedding providers have different embedding methods for documents (to be searched over) vs queries (the search query itself). +For specifics on how to use embedding models, see the [relevant how-to guides here](/docs/how_to/#embedding-models). + ### Vectorstores @@ -477,6 +493,8 @@ const vectorstore = new MyVectorStore(); const retriever = vectorstore.asRetriever(); ``` +For specifics on how to use vector stores, see the [relevant how-to guides here](/docs/how_to/#vectorstores). + ### Retrievers @@ -488,6 +506,8 @@ Retrievers can be created from vector stores, but are also broad enough to inclu Retrievers accept a string query as input and return an array of `Document`s as output. +For specifics on how to use retrievers, see the [relevant how-to guides here](/docs/how_to/#retrievers). + ### Tools @@ -508,6 +528,8 @@ Many agents will only work with tools that have a single string input. Importantly, the name, description, and JSON schema (if used) are all used in the prompt. Therefore, it is really important that they are clear and describe exactly how the tool should be used. You may need to change the default name, description, or JSON schema if the LLM is not understanding how to use the tool. +For specifics on how to use tools, see the [relevant how-to guides here](/docs/how_to/#tools). + ### Toolkits Toolkits are collections of tools that are designed to be used together for specific tasks. They have convenient loading methods. @@ -540,6 +562,7 @@ In order to solve that we built LangGraph to be this flexible, highly-controllab If you are still using AgentExecutor, do not fear: we still have a guide on [how to use AgentExecutor](/docs/how_to/agent_executor). It is recommended, however, that you start to transition to [LangGraph](https://github.com/langchain-ai/langgraphjs). +In order to assist in this we have put together a [transition guide on how to do so](/docs/how_to/migrate_agent). ### Multimodal @@ -549,6 +572,8 @@ Multimodal **outputs** are even less common. As such, we've kept our multimodal In LangChain, most chat models that support multimodal inputs also accept those values in OpenAI's content blocks format. So far this is restricted to image inputs. For models like Gemini which support video and other bytes input, the APIs also support the native, model-specific representations. +For specifics on how to use multimodal models, see the [relevant how-to guides here](/docs/how_to/#multimodal). + ### Callbacks LangChain provides a callbacks system that allows you to hook into the various stages of your LLM application. This is useful for logging, monitoring, streaming, and other tasks. @@ -600,6 +625,8 @@ of the object. If you're creating a custom chain or runnable, you need to remember to propagate request time callbacks to any child objects. +For specifics on how to use callbacks, see the [relevant how-to guides here](/docs/how_to/#callbacks). + ## Techniques ### Function/tool calling @@ -671,6 +698,8 @@ LangChain provides several advanced retrieval types. A full list is below, along | [Multi-Query Retriever](/docs/how_to/multiple_queries/) | Any | Yes | If users are asking questions that are complex and require multiple pieces of distinct information to respond. | This uses an LLM to generate multiple queries from the original one. This is useful when the original query needs pieces of information about multiple topics to be properly answered. By generating multiple queries, we can then fetch documents for each of them. | | [Ensemble](/docs/how_to/ensemble_retriever) | Any | No | If you have multiple retrieval methods and want to try combining them. | This fetches documents from multiple retrievers and then combines them. | +For a high-level guide on retrieval, see this [tutorial on RAG](/docs/tutorials/rag/). + ### Text splitting LangChain offers many different types of `text splitters`. diff --git a/docs/core_docs/docs/how_to/index.mdx b/docs/core_docs/docs/how_to/index.mdx index 4ac9ef722ae8..f3c3db36f7a0 100644 --- a/docs/core_docs/docs/how_to/index.mdx +++ b/docs/core_docs/docs/how_to/index.mdx @@ -47,7 +47,7 @@ These are the core building blocks you can use when building applications. ### Prompt templates -Prompt Templates are responsible for formatting user input into a format that can be passed to a language model. +[Prompt Templates](/docs/concepts/#prompt-templates) are responsible for formatting user input into a format that can be passed to a language model. - [How to: use few shot examples](/docs/how_to/few_shot_examples) - [How to: use few shot examples in chat models](/docs/how_to/few_shot_examples_chat/) @@ -56,7 +56,7 @@ Prompt Templates are responsible for formatting user input into a format that ca ### Example selectors -Example Selectors are responsible for selecting the correct few shot examples to pass to the prompt. +[Example Selectors](/docs/concepts/#example-selectors) are responsible for selecting the correct few shot examples to pass to the prompt. - [How to: use example selectors](/docs/how_to/example_selectors) - [How to: select examples by length](/docs/how_to/example_selectors_length_based) @@ -64,7 +64,7 @@ Example Selectors are responsible for selecting the correct few shot examples to ### Chat models -Chat Models are newer forms of language models that take messages in and output a message. +[Chat Models](/docs/concepts/#chat-models) are newer forms of language models that take messages in and output a message. - [How to: do function/tool calling](/docs/how_to/tool_calling) - [How to: get models to return structured output](/docs/how_to/structured_output) @@ -76,7 +76,7 @@ Chat Models are newer forms of language models that take messages in and output ### LLMs -What LangChain calls LLMs are older forms of language models that take a string in and output a string. +What LangChain calls [LLMs](/docs/concepts/#llms) are older forms of language models that take a string in and output a string. - [How to: cache model responses](/docs/how_to/llm_caching) - [How to: create a custom LLM class](/docs/how_to/custom_llm) @@ -85,7 +85,7 @@ What LangChain calls LLMs are older forms of language models that take a string ### Output parsers -Output Parsers are responsible for taking the output of an LLM and parsing into more structured format. +[Output Parsers](/docs/concepts/#output-parsers) are responsible for taking the output of an LLM and parsing into more structured format. - [How to: use output parsers to parse an LLM response into structured format](/docs/how_to/output_parser_structured) - [How to: parse JSON output](/docs/how_to/output_parser_json) @@ -94,7 +94,7 @@ Output Parsers are responsible for taking the output of an LLM and parsing into ### Document loaders -Document Loaders are responsible for loading documents from a variety of sources. +[Document Loaders](/docs/concepts/#document-loaders) are responsible for loading documents from a variety of sources. - [How to: load CSV data](/docs/how_to/document_loader_csv) - [How to: load data from a directory](/docs/how_to/document_loader_directory) @@ -105,7 +105,7 @@ Document Loaders are responsible for loading documents from a variety of sources ### Text splitters -Text Splitters take a document and split into chunks that can be used for retrieval. +[Text Splitters](/docs/concepts/#text-splitters) take a document and split into chunks that can be used for retrieval. - [How to: recursively split text](/docs/how_to/recursive_text_splitter) - [How to: split by character](/docs/how_to/character_text_splitter) @@ -114,20 +114,20 @@ Text Splitters take a document and split into chunks that can be used for retrie ### Embedding models -Embedding Models take a piece of text and create a numerical representation of it. +[Embedding Models](/docs/concepts/#embedding-models) take a piece of text and create a numerical representation of it. - [How to: embed text data](/docs/how_to/embed_text) - [How to: cache embedding results](/docs/how_to/caching_embeddings) ### Vector stores -Vector stores are databases that can efficiently store and retrieve embeddings. +[Vector stores](/docs/concepts/#vectorstores) are databases that can efficiently store and retrieve embeddings. - [How to: create and query vector stores](/docs/how_to/vectorstores) ### Retrievers -Retrievers are responsible for taking a query and returning relevant documents. +[Retrievers](/docs/concepts/#retrievers) are responsible for taking a query and returning relevant documents. - [How to: use a vector store to retrieve data](/docs/how_to/vectorstore_retriever) - [How to: generate multiple queries to retrieve data for](/docs/how_to/multiple_queries) @@ -148,7 +148,7 @@ Indexing is the process of keeping your vectorstore in-sync with the underlying ### Tools -LangChain Tools contain a description of the tool (to pass to the language model) as well as the implementation of the function to call). +LangChain [Tools](/docs/concepts/#tools) contain a description of the tool (to pass to the language model) as well as the implementation of the function to call. - [How to: create custom tools](/docs/how_to/custom_tools) - [How to: use built-in tools and built-in toolkits](/docs/how_to/tools_builtin) @@ -168,6 +168,8 @@ For in depth how-to guides for agents, please check out [LangGraph](https://lang ### Callbacks +[Callbacks](/docs/concepts/#callbacks) allow you to hook into the various stages of your LLM application's execution. + - [How to: pass in callbacks at runtime](/docs/how_to/callbacks_runtime) - [How to: attach callbacks to a module](/docs/how_to/callbacks_attach) - [How to: pass callbacks into a module constructor](/docs/how_to/callbacks_constructor) @@ -204,6 +206,7 @@ These guides cover use-case specific details. ### Q&A with RAG Retrieval Augmented Generation (RAG) is a way to connect LLMs to external sources of data. +For a high-level tutorial on RAG, check out [this guide](/docs/tutorials/rag/). - [How to: add chat history](/docs/how_to/qa_chat_history_how_to/) - [How to: stream](/docs/how_to/qa_streaming/) @@ -214,6 +217,7 @@ Retrieval Augmented Generation (RAG) is a way to connect LLMs to external source ### Extraction Extraction is when you use LLMs to extract structured information from unstructured text. +For a high level tutorial on extraction, check out [this guide](/docs/tutorials/extraction/). - [How to: use reference examples](/docs/how_to/extraction_examples/) - [How to: handle long text](/docs/how_to/extraction_long_text/) @@ -222,6 +226,7 @@ Extraction is when you use LLMs to extract structured information from unstructu ### Chatbots Chatbots involve using an LLM to have a conversation. +For a high-level tutorial on building chatbots, check out [this guide](/docs/tutorials/chatbot/). - [How to: manage memory](/docs/how_to/chatbots_memory) - [How to: do retrieval](/docs/how_to/chatbots_retrieval) @@ -230,6 +235,7 @@ Chatbots involve using an LLM to have a conversation. ### Query analysis Query Analysis is the task of using an LLM to generate a query to send to a retriever. +For a high-level tutorial on query analysis, check out [this guide](/docs/tutorials/query_analysis/). - [How to: add examples to the prompt](/docs/how_to/query_few_shot) - [How to: handle cases where no queries are generated](/docs/how_to/query_no_queries) @@ -241,6 +247,7 @@ Query Analysis is the task of using an LLM to generate a query to send to a retr ### Q&A over SQL + CSV You can use LLMs to do question answering over tabular data. +For a high-level tutorial, check out [this guide](/docs/tutorials/sql_qa/). - [How to: use prompting to improve results](/docs/how_to/sql_prompting) - [How to: do query validation](/docs/how_to/sql_query_checking) @@ -249,8 +256,25 @@ You can use LLMs to do question answering over tabular data. ### Q&A over graph databases You can use an LLM to do question answering over graph databases. +For a high-level tutorial, check out [this guide](/docs/tutorials/graph/). - [How to: map values to a database](/docs/how_to/graph_mapping) - [How to: add a semantic layer over the database](/docs/how_to/graph_semantic) - [How to: improve results with prompting](/docs/how_to/graph_prompting) - [How to: construct knowledge graphs](/docs/how_to/graph_constructing) + +## [LangGraph.js](https://langchain-ai.github.io/langgraphjs) + +LangGraph.js is an extension of LangChain aimed at +building robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph. + +LangGraph.js documentation is currently hosted on a separate site. +You can peruse [LangGraph.js how-to guides here](https://langchain-ai.github.io/langgraphjs/how-tos/). + +## [LangSmith](https://docs.smith.langchain.com/) + +LangSmith allows you to closely trace, monitor and evaluate your LLM application. +It seamlessly integrates with LangChain, and you can use it to inspect and debug individual steps of your chains as you build. + +LangSmith documentation is hosted on a separate site. +You can peruse [LangSmith how-to guides here](https://docs.smith.langchain.com/how_to_guides/). diff --git a/docs/core_docs/docs/tutorials/index.mdx b/docs/core_docs/docs/tutorials/index.mdx index 9be746b04127..49e496e4599c 100644 --- a/docs/core_docs/docs/tutorials/index.mdx +++ b/docs/core_docs/docs/tutorials/index.mdx @@ -28,3 +28,19 @@ New to LangChain or to LLM app development in general? Read this material to qui - [Build an Extraction Chain](/docs/tutorials/extraction) - [Classify text into labels](/docs/tutorials/classification) - [Summarize text](/docs/tutorials/summarization) + +### LangGraph.js + +LangGraph.js is an extension of LangChain aimed at +building robust and stateful multi-actor applications with LLMs by modeling steps as edges and nodes in a graph. + +LangGraph.js documentation is currently hosted on a separate site. +You can peruse [LangGraph.js tutorials here](https://langchain-ai.github.io/langgraphjs/tutorials/). + +### LangSmith + +LangSmith allows you to closely trace, monitor and evaluate your LLM application. +It seamlessly integrates with LangChain, and you can use it to inspect and debug individual steps of your chains as you build. + +LangSmith documentation is hosted on a separate site. +You can peruse [LangSmith tutorials here](https://docs.smith.langchain.com/tutorials/). diff --git a/docs/core_docs/docs/tutorials/pdf_qa.ipynb b/docs/core_docs/docs/tutorials/pdf_qa.ipynb index 46fbb98a22ff..374437a62730 100644 --- a/docs/core_docs/docs/tutorials/pdf_qa.ipynb +++ b/docs/core_docs/docs/tutorials/pdf_qa.ipynb @@ -117,7 +117,7 @@ "\n", "## Question answering with RAG\n", "\n", - "Next, you'll prepare the loaded documents for later retrieval. Using a [text splitter](/docs/concepts/#text-splitters), you'll split your loaded documents into smaller documents that can more easily fit into an LLM's context window, then load them into a [vector store](/docs/concepts/#vector-stores). You can then create a [retriever](/docs/concepts/#retrievers) from the vector store for use in our RAG chain:\n", + "Next, you'll prepare the loaded documents for later retrieval. Using a [text splitter](/docs/concepts/#text-splitters), you'll split your loaded documents into smaller documents that can more easily fit into an LLM's context window, then load them into a [vector store](/docs/concepts/#vectorstores). You can then create a [retriever](/docs/concepts/#retrievers) from the vector store for use in our RAG chain:\n", "\n", "```{=mdx}\n", "import ChatModelTabs from \"@theme/ChatModelTabs\";\n",