batch calls to LLMs - how are they processed internally?

[sofi444]

Hi there!

I need to pass different instances of a dataset to a LLM in independent calls. I am currently passing them to the model in a loop, like so:

    for idx, instance in tqdm.tqdm(enumerate(data)):
        idx_uuid_map[idx] = instance["id"]
        inputs = {'question':instance["question"],
            'context': instance["context"]}
        if args.output_formatting:
            inputs["format_instructions"] = format_instructions

        response = chain.run(inputs)

I am considering changing it to work with batch calls, like so:

    all_responses = {}
    batch_size = 2
    inputs = []
    for idx, instance in enumerate(data):
        inputs.append({'question':instance["question"],
                                   'context': instance["context"]})
        if len(inputs) == batch_size or len(data)-idx < batch_size:

            batch_responses = chain.batch(inputs)
            inputs = []

            for i, response in enumerate(batch_responses):
                all_responses[idx+i] = response

When using batch calls, are the calls still independent of each other? Verbose=True shows that the chain is entered only once for each batch, which makes sense. I have not set up any memory component for the chain, so I believe the calls should still be independent, but I would like to learn more about how batch() works internally and verify this. TY!

[dosubot[bot]]

🤖

Hello,

Based on the information provided, it seems that you're interested in understanding how the batch() function works in LangChain and whether the batch calls are independent of each other when there is no memory component set up for the chain.

The batch() function in LangChain is designed to handle multiple inputs at once. It takes a list of inputs and an optional configuration. If a maximum concurrency limit (max_concurrency) is not provided, it generates prompts for all inputs at once using the generate_prompt() method and returns the text of the generations. If max_concurrency is provided, it divides the inputs into batches and recursively calls the batch() function on each batch, returning the outputs.

Here is the implementation of the batch() function:

def batch(
        self,
        inputs: List[LanguageModelInput],
        config: Optional[Union[RunnableConfig, List[RunnableConfig]]] = None,
        max_concurrency: Optional[int] = None,
        **kwargs: Any,
    ) -> List[str]:
        config = self._get_config_list(config, len(inputs))

        if max_concurrency is None:
            llm_result = self.generate_prompt(
                [self._convert_input(input) for input in inputs],
                callbacks=[c.get("callbacks") for c in config],
                tags=[c.get("tags") for c in config],
                metadata=[c.get("metadata") for c in config],
                **kwargs,
            )
            return [g[0].text for g in llm_result.generations]
        else:
            batches = [
                inputs[i : i + max_concurrency]
                for i in range(0, len(inputs), max_concurrency)
            ]
            return [
                output
                for batch in batches
                for output in self.batch(batch, config=config, **kwargs)
            ]

As for the independence of batch calls, each call to the chain is independent of the others. The function simply iterates over the list of inputs and applies the chain to each one, without any apparent interaction between different calls. If there is no memory component set up for the chain, there should be no state shared between calls, making them independent.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• libs/langchain/langchain/chains/base.py
• libs/langchain/langchain/llms/base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

Hello,

You're correct in your understanding that the batch() function in LangChain processes multiple inputs by dividing them into batches and processing each batch separately. This is done to optimize the processing speed and manage the load on the system.

When you use the batch() function, each input in the batch is processed independently of the others. This means that the results of processing one input do not affect the results of processing another input in the same batch. This is true even if you have not set up any memory component for the chain.

Here's a simplified explanation of how the batch() function works:

1. If the max_concurrency parameter is not provided, it processes all inputs at once using the generate_prompt method.

2. If max_concurrency is provided, it divides the inputs into batches of size max_concurrency and processes each batch separately using the batch method.

3. The results are then flattened into a single list using a list comprehension.

Here is the relevant code:

if max_concurrency is None:
    llm_result = self.generate_prompt(
        [self._convert_input(input) for input in inputs],
        callbacks=[c.get("callbacks") for c in config],
        tags=[c.get("tags") for c in config],
        metadata=[c.get("metadata") for c in config],
        **kwargs,
    )
    return [g[0].text for g in llm_result.generations]
else:
    batches = [
        inputs[i : i + max_concurrency]
        for i in range(0, len(inputs), max_concurrency)
    ]
    return [
        output
        for batch in batches
        for output in self.batch(batch, config=config, **kwargs)
    ]

This code can be found in the libs/langchain/langchain/llms/base.py file.

I hope this helps! If you have any more questions, feel free to ask.

Sources

• libs/langchain/langchain/llms/base.py
• libs/experimental/langchain_experimental/autonomous_agents/autogpt/prompt_generator.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dariusatmar]

I'm late to the party but I had the same question and thought I'd post what I found for future searchers.

The core concept is the Runnable protocol, which allows most langchain components to execute the various invoke methods in a standardized way.
When you call chain.batch() you really invoke the batch function in Runnables.
You can see that in the function, if the batch length > 1, it calls a get_executor_for_config() which implements a ThreadPoolExecutor based on the max_concurrency config param.

[geeksid1800]

Is there any obvious advantage to using batch() rather than invoke() after iterating over each input? Apart from some lines of code being saved....
The bot saying "The function simply iterates over the list of inputs and applies the chain to each one, without any apparent interaction between different calls" makes me think not...

[pchowdhry]

Does the batch function guarantee the response array will be in the same order as the input array? I've got a set of documents with input data (input_1.txt, input_2.txt,...). I need to write out the response as Input_1_response.txt, Input_2_response.txt.

Is there a way to pass "metadata" into the batch function so that the response object has that metadata in it, so i could parse it an retrieve the filename?

Thanks in advance

-pankaj

[Yen444]

@dosu-bot, @eyurtsev
I am trying to run parallel inputs with batch() methods. I have 2 different chains as follows: one chain has a reranker and the other not. As I observe, the batch method works perfectly for the chain without the reranker but it doesn't work for the chain with the reranker. I don't see the reason why. Does anyone of you know why and how to fix the problem?

Many thanks in advance :)

Reranker

import torch
from FlagEmbedding import FlagReranker

reranker = FlagReranker('BAAI/bge-reranker-large')

Reranker runnale

from langchain_core.runnables import chain as as_runnable

@as_runnable
def bge_reranker_transform(inputs:dict)-> dict:
query = inputs['question']
contexts = inputs['contexts']
...........
return {'question': query, 'contexts': docs}

Rag chains

rag_chain_1 = ({"contexts": retriever, "question": RunnablePassthrough()} | QA_PROMPT | llm)
rag_chain_2 = ({"contexts": retriever, "question": RunnablePassthrough()} | bge_reranker_transform | QA_PROMPT | llm)

[didmar]

Does the batch function guarantee the response array will be in the same order as the input array? I've got a set of documents with input data (input_1.txt, input_2.txt,...). I need to write out the response as Input_1_response.txt, Input_2_response.txt.

Is there a way to pass "metadata" into the batch function so that the response object has that metadata in it, so i could parse it an retrieve the filename?

Thanks in advance

-pankaj

@pchowdhry I just stumbled upon the same issue.
By any chance, did you find a workaround other than passing metadata?

[eyurtsev]

@pchowdhry

Does the batch function guarantee the response array will be in the same order as the input array?

Yes

Is there a way to pass "metadata" into the batch function so that the response object has that metadata in it, so i could parse it an retrieve the filename?

You can pass a list of configs to batch, and then access that information from the implementations, but you shouldn't need to because the response is in the same order as the input.

[Mokoghost]

h, and then access that information from the implementations, but you shouldn't need to because the response is in the same order as the input.

Well, practice shows that it's possible that the response is not in the same order as the input, like 0.1% of the input, when I use async with semaphore to improve invoke efficiency(I don't know why). So we really need this metadata be passed in.