vertexai invalidarguments error when invoking runnablewithmessagehistory

[CLozy]

@dosu

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

prompt = ChatPromptTemplate.from_messages(
                [
                    ("system", "You are a helpful assistant."),
                    ("placeholder", "{history}"),
                    ("human", "{input}"),
                    ("placeholder", "{agent_scratchpad}")
                ]
            )

            chain = prompt | llm
            with_message_history = RunnableWithMessageHistory(
                chain,
                lambda: get_chat_history(strategy_id),
                input_messages_key="input",
                history_messages_key="history",
            )
            try:
                result = await with_message_history.ainvoke(
                    {"input": message},
                    config={"configurable": {"session_id": strategy_id}},
                )

Description

I can successfully invoke the chain but when I trying to invoke my chain with runnable with message history and its throwing this error
400 Unable to submit request because it must include at least one parts field, which describes the prompt input..
What am i missing here

System Info

langchain==0.3.4
langchain-community==0.3.3
langchain-core==0.3.12
langchain-experimental==0.3.2
langchain-google-vertexai==2.0.5
langchain-text-splitters==0.3.0
llama-index-embeddings-langchain==0.2.1
opentelemetry-instrumentation-langchain==0.33.2

Python 3.11.6

platform(mac)

[CLozy]

@dosu , could you look at my issue above

[dosubot[bot]]

The error you're encountering, "InvalidArgument: 400 Unable to submit request because it must include at least one parts field, which describes the prompt input," suggests that the request to VertexAI is missing a required "parts" field. This field is crucial for describing the prompt input that VertexAI expects.

Here are a few things to check:

1. Ensure Proper Placeholder Replacement: Double-check that all placeholders in your ChatPromptTemplate are being correctly replaced with actual values. If any placeholder remains unreplaced, it might lead to missing input data in the request.

2. Verify Message History Format: Make sure that the message history returned by get_chat_history is in the correct format expected by LangChain. Each message should be a BaseMessage object, such as HumanMessage or AIMessage, with the necessary fields.

3. Check Configuration: Ensure that the configuration passed to ainvoke includes all necessary fields required by VertexAI. The error message suggests that a "parts" field is missing, so verify that your configuration aligns with VertexAI's requirements.

4. Review Prompt Construction: Ensure that the prompt is constructed correctly and that all necessary parts are included. This includes checking that the prompt is not missing any required sections or fields.

By addressing these areas, you should be able to resolve the issue with the missing "parts" field in your request to VertexAI.

To continue talking to Dosu, mention @dosu.

[CLozy]

@dosu any idea what kind of prompt input vertexai expects

[dosubot[bot]]

VertexAI expects prompt input in the form of HumanMessage objects, which contain the content of the message as a string. For example, you can create a simple prompt input like this:

message = HumanMessage(content="Hello")

This HumanMessage is then passed to the invoke method of the ChatVertexAI model to generate a response:

response = model.invoke([message])

The prompt input can also include a SystemMessage for context or additional HumanMessage and AIMessage objects to represent a conversation history. The ChatVertexAI model processes these messages by parsing them into a chat history, where SystemMessage can provide context, and AIMessage and HumanMessage represent the conversation flow ^[1][2].

To continue talking to Dosu, mention @dosu.

[CLozy]

@dosu how can i print the request sent to vertexai from the runnablewithmessagehistory

[dosubot[bot]]

To print the request sent to VertexAI from RunnableWithMessageHistory, you can use logging or debugging techniques to capture the request details. Here's how you can do it:

1. Modify the Code to Print the Request: You can add print statements or logging within the RunnableWithMessageHistory or just before the request is sent to VertexAI. This will help you see the exact payload being sent.

2. Use a Custom Wrapper: Wrap the ainvoke method to log the input before it's sent:

   async def log_and_invoke(input_data, config):
       print("Request Input:", input_data)
       print("Request Config:", config)
       result = await with_message_history.ainvoke(input_data, config=config)
       return result
   
   # Use this function instead of directly calling ainvoke
   result = await log_and_invoke({"input": message}, config={"configurable": {"session_id": strategy_id}})

3. Enable Verbose or Debug Mode: If LangChain supports verbose or debug modes, enabling these can help log detailed information about the requests and responses.

By implementing one of these methods, you should be able to capture and inspect the request being sent to VertexAI, allowing you to troubleshoot any issues with the request format or content.

To continue talking to Dosu, mention @dosu.

[CLozy]

@dosu how can i ensure my agent executor always invoke the tools and not just answer from previous queries in history , my code

llm_with_tools = llm.bind_tools(tools)

            tooling_agent = create_tool_calling_agent(llm_with_tools, tools,prompt)

            agent_executor = AgentExecutor(agent=tooling_agent, tools=tools, verbose=True, handle_parsing_errors=True)
           
            with_message_history = RunnableWithMessageHistory(
                agent_executor,
                lambda: get_chat_history(strategy_id),
                input_messages_key="input",
                history_messages_key="history",
            )
            
            try:
                result = await with_message_history.ainvoke(
                    {"input": message, "strategy_id": strategy_id},
                    config={"configurable": {"session_id": strategy_id}},
                )
        
                assistant_response = result["output"]

[dosubot[bot]]

Here's how you can convert your prompt to follow the format of the 'hwchase17/openai-tools-agent' prompt:

PREFIX = """You are a helpful Wheel assistant that answers user questions on Wheel as a product and its uses. You always use tools provided to answer performance or product data queries.

If the question is about performance data like "What is my ROAS?", "How many orders did I get?", "What are my top selling products?", "campaigns performances", "reports data", use the query_dataframe tool and pass this strategy_id {strategy_id}. Also give a little explanation, don't give responses like ROAS can't be calculated or is undefined or is infinite (inf), this is a better answer, "Your ROAS is 0. This means that you haven't generated any revenue from your ads during the time period", and if its time period data is in input, like 7 days or 30 days, use current pd.to_datetime('today'), to calculate.

If it's related to product information on Wheel, like FAQs example "How do I start with Wheel?", "What is the minimum advertising budget I can spend on Wheel", "What are growth recipes", and questions on privacy policy, terms of use and cookie policy, you must use the query_pinecone tool.

If the question is out of scope of tools, answer with your general knowledge.

TOOLS:
------

Assistant has access to the following tools:"""

FORMAT_INSTRUCTIONS = """RESPONSE FORMAT INSTRUCTIONS
----------------------------

When responding to me, please output a response in one of two formats:

**Option 1:**
Use this if you want the human to use a tool.
Markdown code snippet formatted in the following schema:

```json
{{{{
    "action": string, \\\\ The action to take. Must be one of {tool_names}
    "action_input": string \\\\ The input to the action
}}}}

Option #2:
Use this if you want to respond directly to the human. Markdown code snippet formatted in the following schema:

{{{{
    "action": "Final Answer",
    "action_input": string \\\\ You should put what you want to return to use here
}}}}
```"""

SUFFIX = """TOOLS
------
Assistant can ask the user to use tools to look up information that may be helpful in answering the users original question. The tools the human can use are:

{{tools}}

{format_instructions}

USER'S INPUT
--------------------
Here is the user's input (remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else):

{{{{input}}}}"""

This format ensures that your prompt aligns with the 'hwchase17/openai-tools-agent' structure, guiding the agent to decide when to use a tool and how to format its responses ^[1][2][3].

To continue talking to Dosu, mention @dosu.

[CLozy]

@dosu how can you pack that into a chatprompttemplate , modify my prompt code

[dosubot[bot]]

To pack your prompt into a ChatPromptTemplate and modify your existing prompt code, you can use the ChatPromptTemplate class from the LangChain framework. Here's how you can do it:

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.messages import HumanMessage, SystemMessage

# Define your prompt using ChatPromptTemplate
prompt = ChatPromptTemplate([
    ("system", "You are a helpful Wheel assistant that answers user questions on Wheel as a product and its uses. You always use tools provided to answer performance or product data queries. "
               "If the question is about performance data like 'What is my ROAS?', 'How many orders did I get?', 'What are my top selling products?', 'campaigns performances', 'reports data', use the query_dataframe tool and pass this strategy_id {strategy_id}. "
               "Also give a little explanation, don't give responses like ROAS can't be calculated or is undefined or is infinite (inf), this is a better answer, 'Your ROAS is 0. This means that you haven't generated any revenue from your ads during the time period', and if its time period data is in input, like 7 days or 30 days, use current pd.to_datetime('today'), to calculate. "
               "If it's related to product information on Wheel, like FAQs example 'How do I start with Wheel?', 'What is the minimum advertising budget I can spend on Wheel', 'What are growth recipes', and questions on privacy policy, terms of use and cookie policy, you must use the query_pinecone tool. "
               "If the question is out of scope of tools, answer with your general knowledge."),
    MessagesPlaceholder("history"),
    ("human", "{input}"),
    MessagesPlaceholder("agent_scratchpad")
])

# Example of how to invoke the prompt
prompt_value = prompt.invoke(
    {
        "history": [HumanMessage(content="How do I start with Wheel?")],
        "input": "What is my ROAS?",
        "agent_scratchpad": "",
        "strategy_id": "your_strategy_id_here"
    }
)

# The output will be a ChatPromptValue with formatted messages
# You can print or use the prompt_value as needed

This code snippet demonstrates how to create a ChatPromptTemplate with a system message, placeholders for chat history and agent scratchpad, and a human message with a variable. You can then invoke the template with the necessary input to get the formatted chat prompt ^[1][2].

To continue talking to Dosu, mention @dosu.

[CLozy]

@dosu i meant modify the code to follow the format of the 'hwchase17/openai-tools-agent'

[dosubot[bot]]

To modify your code to follow the format of the 'hwchase17/openai-tools-agent', you can structure your ChatPromptTemplate to include the necessary system message, format instructions, and placeholders. Here's how you can do it:

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.messages import SystemMessage, HumanMessage

# Define the system message and format instructions
SYSTEM_MESSAGE_PREFIX = """You are a helpful Wheel assistant that answers user questions on Wheel as a product and its uses. You always use tools provided to answer performance or product data queries.

If the question is about performance data like "What is my ROAS?", "How many orders did I get?", "What are my top selling products?", "campaigns performances", "reports data", use the query_dataframe tool and pass this strategy_id {strategy_id}. Also give a little explanation, don't give responses like ROAS can't be calculated or is undefined or is infinite (inf), this is a better answer, "Your ROAS is 0. This means that you haven't generated any revenue from your ads during the time period", and if its time period data is in input, like 7 days or 30 days, use current pd.to_datetime('today'), to calculate.

If it's related to product information on Wheel, like FAQs example "How do I start with Wheel?", "What is the minimum advertising budget I can spend on Wheel", "What are growth recipes", and questions on privacy policy, terms of use and cookie policy, you must use the query_pinecone tool.

If the question is out of scope of tools, answer with your general knowledge.

TOOLS:
------

Assistant has access to the following tools:"""

FORMAT_INSTRUCTIONS = """The way you use the tools is by specifying a json blob.
Specifically, this json should have an `action` key (with the name of the tool to use) and an `action_input` key (with the input to the tool going here).

The only values that should be in the "action" field are: {tool_names}

The $JSON_BLOB should only contain a SINGLE action, do NOT return a list of multiple actions. Here is an example of a valid $JSON_BLOB:

{{{{
"action": $TOOL_NAME,
"action_input": $INPUT
}}}}

ALWAYS use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action:

$JSON_BLOB

Observation: the result of the action
... (this Thought/Action/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question"""

SYSTEM_MESSAGE_SUFFIX = """Begin! Reminder to always use the exact characters `Final Answer` when responding."""

# Create the ChatPromptTemplate
prompt = ChatPromptTemplate([
    SystemMessage(content=SYSTEM_MESSAGE_PREFIX),
    MessagesPlaceholder("history"),
    HumanMessage(content="{input}"),
    MessagesPlaceholder("agent_scratchpad")
])

# Example of how to invoke the prompt
prompt_value = prompt.invoke(
    {
        "history": [HumanMessage(content="How do I start with Wheel?")],
        "input": "What is my ROAS?",
        "agent_scratchpad": "",
        "strategy_id": "your_strategy_id_here"
    }
)

# The output will be a ChatPromptValue with formatted messages
# You can print or use the prompt_value as needed

This setup aligns your prompt with the 'hwchase17/openai-tools-agent' format, ensuring that the agent uses the tools appropriately and follows the specified response structure ^[1][2][3].

To continue talking to Dosu, mention @dosu.