How to write a document section based on the writing style of some documents but taking into account data from research papers?

[levalencia]

I have a business case that I am not sure how to start, so looking for some community feedback.
We want to be able to ask the chatbot to generate a section of the document called "State of the art", based on the writing style of previous documents (around 25 documents in blob storage). However the data for that section must come from some PDFS (research papers) which are in another blob storage container.

For now, what I did its to index in Pinecone all those documents into 2 different namespaces, but whats next in terms of chains? I have no clue yet.

[dosubot[bot]]

Answer generated by a 🤖

Answer

Based on the information you've provided and the capabilities of LangChain, here's a high-level approach you could take:

1. Extract data from PDFs: You can use the AzureBlobStorageContainerLoader class to load documents from your Azure Blob Storage container, and then use a PDF parser such as PDFMinerParser to parse the loaded PDF documents. Here's a sample code snippet:

   from libs.langchain.langchain.document_loaders.azure_blob_storage_container import AzureBlobStorageContainerLoader
   from langchain.document_loaders.parsers.pdf import PDFMinerParser
   
   # Initialize the loader with the connection string, container name, and blob prefix
   loader = AzureBlobStorageContainerLoader(conn_str="<your_connection_string>", container="<your_container_name>", prefix="<your_blob_prefix>")
   
   # Load the documents
   documents = loader.load()
   
   # Initialize the PDF parser
   parser = PDFMinerParser()
   
   # Now you can process the loaded documents
   for document in documents:
       # Parse the document
       parsed_document = parser.parse(document)
       # Your processing logic here

2. Index documents in Pinecone: You've already done this step, but just to reiterate, you can use the PineconeHybridSearchRetriever class to index and retrieve documents in Pinecone.

3. Analyze and extract the writing style: This is a complex task that may require a combination of techniques. Unfortunately, I couldn't find a specific feature in LangChain that directly supports this. You might need to use external libraries or services to analyze the writing style of your documents.

4. Generate text based on a specific writing style: You can use the PromptTemplate class to generate text based on a specific writing style. Here's a sample code snippet:

   class MyWritingStylePromptTemplate(BasePromptTemplate):
       input_variables = ["variable1", "variable2"]
   
       def format_prompt(self, **kwargs: Any) -> PromptValue:
           # Format the text in your specific writing style
           formatted_text = f"My writing style: {kwargs['variable1']} and {kwargs['variable2']}"
           return PromptValue(formatted_text)

5. Combine the writing style with the data: This is another complex task that may require a combination of techniques. Unfortunately, I couldn't find a specific feature in LangChain that directly supports this. You might need to develop a custom solution that combines the writing style extracted from your documents with the data extracted from your research papers to generate the "State of the art" section.

Please note that this is a high-level approach and you may need to adjust it based on your specific requirements and constraints. If you need more detailed guidance or if you have any other questions, please let me know!

Sources

• libs/langchain/langchain/vectorstores/pinecone.py
• docs/extras/integrations/providers/pinecone.mdx
• libs/langchain/langchain/retrievers/pinecone_hybrid_search.py
• libs/langchain/langchain/document_loaders/azure_blob_storage_container.py
• libs/langchain/langchain/document_loaders/parsers/init.py
• libs/langchain/langchain/schema/prompt_template.py
• libs/langchain/tests/unit_tests/prompts/test_prompt.py
• libs/langchain/langchain/evaluation/qa/generate_prompt.py

---

This response is meant to be useful, save you time, and share context. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.