vertexai[patch]: enable agents for vertex

[ccurme]

from langchain.agents import AgentExecutor, create_tool_calling_agent, tool
from langchain_google_vertexai import ChatVertexAI
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder


prompt = ChatPromptTemplate.from_messages(
    [
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad"),
    ]
)
model = ChatVertexAI(model_name="gemini-pro", temperature=0)


@tool
def magic_function(input: int) -> str:
    """Applies a magic function to an input."""
    return str(input + 2)



tools = [magic_function]

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input": "what is the value of magic_function(3)?"})

> Entering new AgentExecutor chain...

Invoking: `magic_function` with `{'input': 3.0}`


5The value of magic_function(3) is {"content": "5"}.

> Finished chain.
{'input': 'what is the value of magic_function(3)?',
 'output': 'The value of magic_function(3) is {"content": "5"}.'}

[lkuligin]

nits: I believe this else block is not needed

[ccurme]

@lkuligin @baskaryan @efriis we have at least two options to make this work with agents. The underlying issue is that Langchain requires tool_call_ids on instances of ToolMessage (i.e., when passing the result of a tool execution back to the model).

Following the example in the vertex docs here, from what I can tell tool calls and responses are identified with a "name" attribute:

role: "model"
parts {
  function_call {
    name: "get_current_weather"
    args {
      fields {
        key: "location"
        value {
          string_value: "boston"
        }
      }
    }
  }
}

parts {
  function_response {
    name: "get_current_weather"
    response {
      fields {
        key: "content"
        value {
          string_value: "response"
        }
      }
    }
  }
}

Option 1 (implemented here): generate IDs on tool invocations. Then when processing the ToolMessage, we fetch the name from the most recent tool invocation.

Option 2 (implemented locally): use the tool name as the ID. This "just works", but tool calls may look funny with id and name the same. Let me know what you think.