How do I integrate ChatOpenAI, UpstashRedisChatMessageHistory and tool calling?

[henriquegdantas]

Here's an overview of my current implementation and below the error I am getting everytime the AI calls a tool. How do I fix this tool calling? I DON'T WANT TO USE LANGGRAPH. There must be a solution using LCEL, for example.

const systemPrompt = '...';
const tools = ['handoff'] as DynamicTool[];
const model = new ChatOpenAI({ openAIApiKey: env.OPENAI_API_KEY }).bindTools(tools).withRetry({
stopAfterAttempt: 3,
});

const prompt = ChatPromptTemplate.fromMessages([
['system', systemPrompt],
new MessagesPlaceholder('history'),
['human', '{input}'],
]);

const chain = prompt.pipe(model);

const chainWithHistory = new RunnableWithMessageHistory({
  runnable: chain,
  getMessageHistory: (sessionId: string) =>
    new UpstashRedisChatMessageHistory({
      sessionId,
      sessionTTL: 30 * 24 * 60 * 60, // 30 days
      config: {
        url: env.UPSTASH_REDIS_URL,
        token: env.UPSTASH_REDIS_TOKEN,
      },
    }),
  inputMessagesKey: 'input',
  historyMessagesKey: 'history',
});

const response = await chainWithHistory.invoke(
{
input: '...',
},
{
configurable: {
sessionId: history:${userKey},
},
},
);

console.log(ansis.green('OpenAI responded:'), response.content);

---

Result: {
answer: undefined,
error: BadRequestError: 400 An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_2dXaMrjN7sZeyMJ3v23CA5kx

Troubleshooting URL: https://js.langchain.com/docs/troubleshooting/errors/INVALID_TOOL_RESULTS/

  at Function.generate (/home/[redacted]/projects/[redacted]/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/error.ts:72:14)
  at OpenAI.makeStatusError (/home/[redacted]/projects/[redacted]/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/core.ts:435:21)
  at OpenAI.makeRequest (/home/[redacted]/projects/[redacted]/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/core.ts:499:24)
  at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
  at async /home/[redacted]/projects/[redacted]/node_modules/.pnpm/@[email protected]_@[email protected][email protected][email protected]__/node_modules/@langchain/openai/dist/chat_models.cjs:1548:29
  at async RetryOperation._fn (/home/[redacted]/projects/[redacted]/node_modules/.pnpm/[email protected]/node_modules/p-retry/index.js:50:12) {
status: 400,
headers: {
  'access-control-expose-headers': 'X-Request-ID',
  'cf-cache-status': 'DYNAMIC',
  'cf-ray': '8f38bc669895f1de-GRU',
  connection: 'keep-alive',
  'content-length': '325',
  'content-type': 'application/json',
  date: 'Tue, 17 Dec 2024 17:47:01 GMT',
  'openai-organization': '[redacted]',
  'openai-processing-ms': '48',
  'openai-version': '2020-10-01',
  server: 'cloudflare',
  'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
  vary: 'Accept-Encoding',
  'x-content-type-options': 'nosniff',
  'x-ratelimit-limit-requests': '10000',
  'x-ratelimit-limit-tokens': '10000000',
  'x-ratelimit-remaining-requests': '9999',
  'x-ratelimit-remaining-tokens': '9998421',
  'x-ratelimit-reset-requests': '6ms',
  'x-ratelimit-reset-tokens': '9ms',
  'x-request-id': 'req_434eb1886d02bb66398ad667afff0527'
},
request_id: 'req_434eb1886d02bb66398ad667afff0527',
error: {
  message: "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'. The following tool_call_ids did not have response messages: call_2dXaMrjN7sZeyMJ3v23CA5kx",
  type: 'invalid_request_error',
  param: 'messages.[53].role',
  code: null
},
code: null,
param: 'messages.[53].role',
type: 'invalid_request_error',
lc_error_code: 'INVALID_TOOL_RESULTS',
attemptNumber: 3,
retriesLeft: 0

}

I even tried adding a new MessagesPlaceholder('agent_scratchpad') in my ChatPromptTemplate.fromMessages array, but I get:

Error: Missing value for input variable agent_scratchpad
Troubleshooting URL: https://js.langchain.com/docs/troubleshooting/errors/INVALID_PROMPT_INPUT/
lc_error_code: 'INVALID_PROMPT_INPUT'

[dosubot[bot]]

To resolve the tool calling issue in your implementation, ensure that each tool call is followed by a corresponding response message. Here's how you can address the problem:

1. Capture Tool Responses: After making a tool call, capture the response and append it to the chat history. This ensures that each tool call has a corresponding response message.

2. Update Chat History: Make sure to update the chat history with both the tool call and its response before invoking the model again. This maintains the correct sequence of messages.

3. Example Implementation:

   const toolResponse = await dummyTool.invoke(responseMessage.tool_calls![0]);
   chatHistory.push(responseMessage);
   chatHistory.push(toolResponse);
   await modelWithTools.invoke(chatHistory);

4. Ensure One-to-One Correspondence: For each tool call, there should be a matching response. If using multiple tools, repeat this process for each tool call.

Regarding the agent_scratchpad error, ensure that you provide a value for this variable in your prompt template. If it's not needed, you can remove it from the ChatPromptTemplate.fromMessages array to avoid the "missing value" error ^[1][2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[henriquegdantas]

@dosu, I am calling the chain only once, I though RunnableWithMessageHistory handled saving context to memory automatically.

Do I really have to save tool responses manually? If so, how do I do this with UpstashRedisChatMessageHistory? It only has these options:

• BaseListChatMessageHistory.addAIMessage(message: string): Promise
• UpstashRedisChatMessageHistory.addMessage(message: BaseMessage): Promise
• BaseListChatMessageHistory.addMessages(messages: BaseMessage[]): Promise
• BaseListChatMessageHistory.addUserMessage(message: string): Promise

Also, BaseMessage doesn't seem to allow me to set a role for the message I am inserting so I could use the 'tool' role.

[dosubot[bot]]

The RunnableWithMessageHistory does not automatically call functions; you need to handle this manually. Here's how you can call functions defined inside DynamicTool and DynamicStructuredTool, and pass the function call result back to the AI:

1. Invoke the LLM with a query to get the tool calls:

   query = "What is 3 * 12? Also, what is 11 + 49?"
   ai_msg = llm_with_tools.invoke(query)

2. Iterate over the tool calls in the AI message and execute the corresponding functions:

   from langchain_core.messages import ToolMessage
   
   messages = [HumanMessage(query)]
   messages.append(ai_msg)
   for tool_call in ai_msg.tool_calls:
       selected_tool = {"add": add, "multiply": multiply}[tool_call["name"].lower()]
       tool_output = selected_tool.invoke(tool_call["args"])
       messages.append(ToolMessage(tool_output, tool_call_id=tool_call["id"]))

3. Pass the tool results back to the AI by invoking the LLM with the updated messages:

   llm_with_tools.invoke(messages)

This process involves invoking the LLM to get tool calls, executing the functions manually, and then passing the results back to the AI using ToolMessages ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[henriquegdantas]

@dosu, I am using typescript, so the type definitions are different. The AI response goes into the 'content' property of the 'ToolMessage' constructor?

[dosubot[bot]]

Yes, in TypeScript, the AI response should be placed in the 'content' property of the 'ToolMessage' constructor. The ToolMessage includes the role and content properties, where content is used to store the result or response ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[henriquegdantas]

@dosu, what about parallel function calling? Can I add two or more ToolMessages or somehow it goes into a single entry?
Also help me diagnose this error:

const systemPrompt = '...';
const tools = ['handoff'] as DynamicTool[];
const model = new ChatOpenAI({ openAIApiKey: env.OPENAI_API_KEY }).bindTools(tools).withRetry({
stopAfterAttempt: 3,
});

const prompt = ChatPromptTemplate.fromMessages([
['system', systemPrompt],
new MessagesPlaceholder('history'),
['human', '{input}'],
]);

const chain = prompt.pipe(model);
const chainWithHistory = new RunnableWithMessageHistory({
runnable: chain,
getMessageHistory: (sessionId: string) =>
new UpstashRedisChatMessageHistory({
sessionId,
sessionTTL: 30 * 24 * 60 * 60, // 30 days
config: {
url: env.UPSTASH_REDIS_URL,
token: env.UPSTASH_REDIS_TOKEN,
},
}),
inputMessagesKey: 'input',
historyMessagesKey: 'history',
});
const response = await chainWithHistory.invoke({input: '...',},{configurable: {sessionId: history:${userKey},},},);
if (response.tool_calls?.length) {
console.log('Tool calls in response:', response.tool_calls);

  for (const toolCall of response.tool_calls) {
    const tool = tools.find(x => x.name === toolCall.name);

    let status = 'success' as 'success' | 'error';
    let content: string = '';

    if (!tool) {
      console.error('CALLED_TOOL_NOT_FOUND', toolCall.name);
      status = 'error';
      content = 'CALLED_TOOL_NOT_FOUND';
    } else {
      const result = (await tool.func(
        JSON.stringify(toolCall.args),
      )) as string;

      content = result;
      status = 'success';
    }

    console.log('Tool call result:', { status, content });

    const toolMessage = new ToolMessage({
      tool_call_id: toolCall.id!,
      content,
      status,
    });

    await history.addMessage(toolMessage);
    console.log('Added tool call to history:', toolCall);
  }
} else {
  console.log('No tool calls in response');
}

---

Result: {
answer: undefined,
error: BadRequestError: 400 Invalid value: 'tool_call'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', and 'audio'.
at Function.generate (/home/henriqued/projects/programy-chatbots-swarm/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/error.ts:72:14)
at OpenAI.makeStatusError (/home/henriqued/projects/programy-chatbots-swarm/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/core.ts:435:21)
at OpenAI.makeRequest (/home/henriqued/projects/programy-chatbots-swarm/node_modules/.pnpm/[email protected][email protected]/node_modules/openai/src/core.ts:499:24)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async /home/henriqued/projects/programy-chatbots-swarm/node_modules/.pnpm/@langchain+openai@0.3.14_@langchain[email protected][email protected][email protected]__/node_modules/@langchain/openai/dist/chat_models.cjs:1548:29
at async RetryOperation._fn (/home/henriqued/projects/programy-chatbots-swarm/node_modules/.pnpm/[email protected]/node_modules/p-retry/index.js:50:12) {
status: 400,

[dosubot[bot]]

You can perform parallel function calling by adding multiple ToolMessages separately. Each ToolMessage corresponds to a single tool call, and you can have multiple tool calls in the tool_calls array of the AIMessage. This allows you to handle multiple tool calls in sequence or parallel, depending on your implementation ^[1].

Regarding the error "Invalid value: 'tool_call'. Supported values are: 'text', 'image_url', 'input_audio', 'refusal', and 'audio'", this indicates that you are using an unsupported value type. Ensure that when you are creating a ToolCall or ToolMessage, you are using the correct type value. The type for a ToolMessage should be "tool", and for a ToolCall, it should be "tool_call". If you are mistakenly using "tool_call" as a type in a context where only 'text', 'image_url', 'input_audio', 'refusal', or 'audio' are supported, you need to adjust your code to use one of these supported types instead ^[1].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}