Using WebResearchRetriever for PDFs as well as HTML

[matthewosm]

Hi I'm using WebResearchRetriever to do materials research, unfortunately the top results are often PDFs so I get the error:

langchain.document_loaders.async_html:Failed to decode content from xyz.pdf

I've seen there is a PDF Loader, but I'm not quite sure how to get it to work with my script.

Here is what I have so far:

import faiss, logging
from langchain.chains import RetrievalQAWithSourcesChain
from langchain.chat_models import ChatOpenAI
from langchain.retrievers.web_research import WebResearchRetriever
from langchain.vectorstores import FAISS 
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.docstore import InMemoryDocstore  
from langchain.utilities import GoogleSearchAPIWrapper

embedding_model = OpenAIEmbeddings()
index = faiss.IndexFlatL2(1536)
vectorstore = FAISS(embedding_model.embed_query, index, InMemoryDocstore({}), {})

llm = ChatOpenAI(model_name="gpt-3.5-turbo-16k", temperature=0.5, streaming=True)

search = GoogleSearchAPIWrapper()   

web_retriever = WebResearchRetriever.from_llm(vectorstore=vectorstore, llm=llm, search=search, num_search_results=6)

question = "Define THIS MATERIAL"

if question:
    logging.basicConfig()
    logging.getLogger("langchain.retrievers.web_research").setLevel(logging.INFO)    
    qa_chain = RetrievalQAWithSourcesChain.from_chain_type(llm, retriever=web_retriever)

    result = qa_chain({"question": question})
    answer, sources = result['answer'], result['sources']
    print(answer, sources)

[dosubot[bot]]

Answer generated by a 🤖

Answer

Hi,

Based on the context provided, it seems like you want to modify the WebResearchRetriever to handle both HTML and PDF documents. To do this, you would need to add a PDF loader and transformer to the class. This would involve importing the necessary libraries to handle PDFs, creating a new PDF loader and transformer, and modifying the _get_relevant_documents method to handle PDFs.

Here's an example of how you might do this:

from langchain.document_loaders import AsyncPdfLoader  # Assuming this exists
from langchain.document_transformers import Pdf2TextTransformer  # Assuming this exists

class WebResearchRetriever(BaseRetriever):
    # ... existing code ...

    def _get_relevant_documents(
        self,
        query: str,
        *,
        run_manager: CallbackManagerForRetrieverRun,
    ) -> List[Document]:
        # ... existing code ...

        # Load, split, and add new urls to vectorstore
        if new_urls:
            html_loader = AsyncHtmlLoader([url for url in new_urls if url.endswith('.html')])
            pdf_loader = AsyncPdfLoader([url for url in new_urls if url.endswith('.pdf')])

            html2text = Html2TextTransformer()
            pdf2text = Pdf2TextTransformer()

            logger.info("Indexing new urls...")
            html_docs = html_loader.load()
            pdf_docs = pdf_loader.load()

            html_docs = list(html2text.transform_documents(html_docs))
            pdf_docs = list(pdf2text.transform_documents(pdf_docs))

            docs = html_docs + pdf_docs
            docs = self.text_splitter.split_documents(docs)
            self.vectorstore.add_documents(docs)
            self.url_database.extend(new_urls)

        # ... existing code ...

In this example, we're assuming that AsyncPdfLoader and Pdf2TextTransformer classes exist in the langchain.document_loaders and langchain.document_transformers modules respectively. These classes would be responsible for loading PDF documents from URLs and converting them to text, similar to how AsyncHtmlLoader and Html2TextTransformer handle HTML documents.

Please note that this is a simplified example and might need adjustments based on the actual implementation of the AsyncPdfLoader and Pdf2TextTransformer classes, as well as how the WebResearchRetriever class is used in your application.

If these classes (AsyncPdfLoader and Pdf2TextTransformer) do not exist in the LangChain library, you would need to implement them yourself or find a suitable library that provides similar functionality.

I hope this helps! Let me know if you have any other questions.

Sources

• libs/langchain/langchain/document_loaders/pdf.py
• libs/langchain/tests/integration_tests/document_loaders/test_pdf.py
• libs/langchain/langchain/retrievers/web_research.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.