Running LangChain for multiple queries simultaneously

[DeviousLab]

Hi all,

I'm currently using Python to try and develop an internal application for a business where it would be able to scrape a document and put each line into a row of a table/dataframe. Then I would be calling gpt-3.5-turbo to cross-reference the value in the row against a vectorstore with the source documents to check whether there is a relationship or not.

There isn't much information related to a topic like this so I'm looking for advice. This is the code I've got so far (reference):

@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
async def async_generate(chain,query):
    if query.isupper() and len(query) <= 6:
        resp = await chain.arun(query=query)
        reply_list.append(resp)
    else: 
        reply_list.append('')
    time.sleep(20)
    print(resp)

@retry(wait=wait_random_exponential(min=1, max=60), stop=stop_after_attempt(6))
async def generate_concurrently():
    llm = ChatOpenAI(temperature=0.0)
    embeddings = OpenAIEmbeddings()
    vectorstore = Pinecone.from_existing_index(
        index_name="foo", embedding=embeddings, namespace="foo"
    )

    template = """foo"""

    PROMPT = PromptTemplate(template=template, input_variables=["context", "question"])
    chain_type_kwargs = {"prompt": PROMPT}
    retriever = vectorstore.as_retriever()
    chain = RetrievalQA.from_chain_type(llm=llm, chain_type="stuff", retriever=retriever, chain_type_kwargs=chain_type_kwargs)

    tasks = [async_generate(chain, query) for query in value_list]
    await asyncio.gather(*tasks)

#     Original async API code---
#     tasks = [async_generate(chain) for _ in range(2)]
#     await asyncio.gather(*tasks)
    
await generate_concurrently()

Are there suggestions to improve the efficiency of the code? Or something that might be implemented in the wrong fashion? Thanks in advance!