What's the difference between passing tools to the Prompt vs the ChatClient?

[rwankar]

One can create a Prompt with tools e.g

Prompt prompt = new Prompt(userText, ToolCallingChatOptions.builder().toolCallbacks(functionCallbacks).build());
or one can pass tools to the ChatClient e.g.

ChatClient client = ...;
client.prompt(text).tools(tools).call()

Is it correct to assume that if tools are passed into the prompt then one can have prompt specific tools and you don't have to pass them separately whereas tools passed to the client directly might be more suitable for prompt independent tools?

[seungy0]

Hello,

As you mentioned, both

Prompt prompt = new Prompt(userText, ToolCallingChatOptions.builder().toolCallbacks(functionCallbacks).build());

and

ChatClient client = ...;
client.prompt(text).tools(tools).call();

work similarly.

However, in DefaultChatClient, if tools are already defined at the ChatClient level, any tools specified in the Prompt are ignored. This behavior can be observed in the prompt(Prompt prompt) method:

@Override
public ChatClientRequestSpec prompt(Prompt prompt) {
    Assert.notNull(prompt, "prompt cannot be null");

    DefaultChatClientRequestSpec spec = new DefaultChatClientRequestSpec(this.defaultChatClientRequest);

    // Options
    if (prompt.getOptions() != null) {
        spec.options(prompt.getOptions());
    }

    // Messages
    if (prompt.getInstructions() != null) {
        spec.messages(prompt.getInstructions());
    }

    return spec;
}

Since tools are not explicitly set from the Prompt in this method, any tools defined at the Prompt level will be ignored if ChatClient already has predefined tools.

This means that:

• If you define tools at the ChatClient level, they will always be used.
• If you want prompt-specific tools, you need to ensure that ChatClient does not have globally defined tools, or modify the client’s behavior accordingly.

You can also check this behavior in the AdvisedRequest.java (Lines 176-189)

[rwankar]

@seungy0 thanks for the reply but I'm not following this comment.

Since tools are not explicitly set from the Prompt in this method, any tools defined at the Prompt level will be ignored if ChatClient already has predefined tools.

The following block suggests that ChatClientRequestSpec will use tools in Prompt (as part of the options).

    // Options
    if (prompt.getOptions() != null) {
        spec.options(prompt.getOptions());
    }

I can see that AdvisedRequest seems to ignore the tools in the Prompt IF tools are already set on AdvisedRequest. May be that allows an Advisor to set tools on the AdvisedRequest and override the tools provided via the Prompt.

So we still have the question about what happens with the tools provided on the ChatClient.