Is there an equivalent of HydeRetriever from LangChain.js in the Python version of LangChain?

[octadion]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

# langchain-js
import { HydeRetriever } from "langchain/retrievers/hyde";

retriever = new HydeRetriever({
        vectorStore: vectorstore,
        llm: streamedModel,
        k: 1,
        callbacks: [
          {
            handleRetrieverEnd(documents) {
              resolveWithDocuments(documents);
            },
          },
        ]
      });

# langchain python?

Description

I've been working with LangChain.js and found the HydeRetriever. However, when I tried to find something similar in the Python version of LangChain, I came across HypotheticalDocumentEmbedder. Are these two equivalent, or is there a different class or method in Python that serves the same purpose as HydeRetriever?

System Info

langchain==0.2.14
langchain-anthropic==0.1.23
langchain-cohere==0.2.3
langchain-community==0.2.12
langchain-core==0.2.35
langchain-experimental==0.0.64
langchain-google-genai==1.0.10
langchain-ollama==0.1.1
langchain-openai==0.1.19
langchain-text-splitters==0.2.2
langchainhub==0.1.21

[dosubot[bot]]

Yes, the HypotheticalDocumentEmbedder class in the Python version of LangChain corresponds to the HydeRetriever from LangChain.js ^[1][2].

To continue talking to Dosu, mention @dosu.

[octadion]

How would I implement this HydeRetriever code in the Python version using HypotheticalDocumentEmbedder?

I'm trying to translate the following LangChain.js code to Python:

retriever = new HydeRetriever({
    vectorStore: vectorstore,
    llm: model,
    k: 1,
    callbacks: [
      {
        handleRetrieverEnd(documents) {
          resolveWithDocuments(documents);
        },
      },
    ]
});

@dosu

[dosubot[bot]]

To implement the HydeRetriever code in Python using HypotheticalDocumentEmbedder, follow these steps:

1. Install the necessary packages:

pip install -U langchain-cli
pip install --upgrade --quiet embedchain
langchain app add hyde

2. Add the following code to your server.py file:

from hyde.chain import chain as hyde_chain

add_routes(app, hyde_chain, path="/hyde")

3. Configure LangSmith (optional):

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=<your-api-key>
export LANGCHAIN_PROJECT=<your-project>  # if not specified, defaults to "default"

4. Start the LangServe instance:

langchain serve

5. Access the template from code:

from langserve.client import RemoteRunnable

runnable = RemoteRunnable("http://localhost:8000/hyde")

6. Implement the EmbedchainRetriever:

from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever
from embedchain import Pipeline

class EmbedchainRetriever(BaseRetriever):
    """`Embedchain` retriever."""

    def __init__(self, client):
        self.client = client

    @classmethod
    def create(cls, yaml_path: Optional[str] = None) -> EmbedchainRetriever:
        """
        Create an EmbedchainRetriever from a YAML configuration file.

        Args:
            yaml_path: Path to the YAML configuration file. If not provided,
                       a default configuration is used.

        Returns:
            An instance of EmbedchainRetriever.
        """
        if yaml_path:
            client = Pipeline.from_config(yaml_path=yaml_path)
        else:
            client = Pipeline()
        return cls(client=client)

    def add_texts(self, texts: Iterable[str]) -> List[str]:
        """Run more texts through the embeddings and add to the retriever.

        Args:
            texts: Iterable of strings/URLs to add to the retriever.

        Returns:
            List of ids from adding the texts into the retriever.
        """
        ids = []
        for text in texts:
            _id = self.client.add(text)
            ids.append(_id)
        return ids

    def _get_relevant_documents(self, query: str) -> List[Document]:
        res = self.client.search(query)
        docs = []
        for r in res:
            docs.append(
                Document(
                    page_content=r["context"],
                    metadata={
                        "source": r["metadata"]["url"],
                        "document_id": r["metadata"]["doc_id"],
                    },
                )
            )
        return docs

7. Add data to the retriever:

retriever = EmbedchainRetriever.create()

retriever.add_texts(
    [
        "https://en.wikipedia.org/wiki/Elon_Musk",
        "https://www.forbes.com/profile/elon-musk",
        "https://www.youtube.com/watch?v=RcYjXbSJBN8",
    ]
)

8. Use the retriever to find relevant documents:

result = retriever.invoke("How many companies does Elon Musk run and name those?")
print(result)

This setup will allow you to use the HydeRetriever with HypotheticalDocumentEmbedder in your Python project ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[Luminilion]

https://github.com/langchain-ai/langchain/blob/master/cookbook/hypothetical_document_embeddings.ipynb