Relevant tables choice with bind_tools

[MartinV279]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from typing import Optional, Type
from pydantic import BaseModel, Field, ConfigDict

from langchain_core.callbacks import (
    CallbackManagerForToolRun,
)

from langchain_core.tools import BaseTool
from langchain_openai import ChatOpenAI
from langchain_community.utilities import SQLDatabase
import requests
import os


url = "https://storage.googleapis.com/benchmarks-artifacts/chinook/Chinook.db"

response = requests.get(url)

if response.status_code == 200:
    # Open a local file in binary write mode
    with open("Chinook.db", "wb") as file:
        # Write the content of the response (the file) to the local file
        file.write(response.content)
    print("File downloaded and saved as Chinook.db")
else:
    print(f"Failed to download the file. Status code: {response.status_code}")

db = SQLDatabase.from_uri("sqlite:///Chinook.db")

class BaseSQLDatabaseTool(BaseModel):
    """Base tool for interacting with a SQL database."""

    db: SQLDatabase = Field(exclude=True)

    model_config = ConfigDict(
        arbitrary_types_allowed=True,
    )


class _QuerySQLDataBaseToolInput(BaseModel):
    query: str = Field(..., description="A detailed and correct SQL query.")

class _InfoSQLDatabaseToolInput(BaseModel):
    table_names: str = Field(
        ...,
        description=(
            "A comma-separated list of the table names for which to return the schema. "
            "Example input: 'table1, table2, table3'"
        ),
    )

class InfoSQLDatabaseTool(BaseSQLDatabaseTool, BaseTool):  # type: ignore[override, override]
    """Tool for getting metadata about a SQL database."""

    name: str = "sql_db_schema"
    description: str = "Get the schema and sample rows for the specified SQL tables."
    args_schema: Type[BaseModel] = _InfoSQLDatabaseToolInput

    def _run(
        self,
        table_names: str,
        run_manager: Optional[CallbackManagerForToolRun] = None,
    ) -> str:
        """Get the schema for tables in a comma-separated list."""
        return self.db.get_table_info_no_throw(
            [t.strip() for t in table_names.split(",")]
        )

tool = InfoSQLDatabaseTool(db=db)

llm = ChatOpenAI(
    model="gpt-4o",
    temperature=0,
    max_tokens=None,
    timeout=None,
    max_retries=2,
)

msg = "Human: Which sales agent made the most in sales in 2009?\nAI: \nTool: Album, Artist, Customer, Employee, Genre, Invoice, InvoiceLine, MediaType, Playlist, PlaylistTrack, Track"

llm.invoke(msg)

# OUTPUT:
# AIMessage(content='To determine which sales agent made the most in sales in 2009, we would need access to the sales data for that year, specifically the records of sales transactions and the associated sales agents. This information would typically be found in a database or a sales record system that includes tables such as Employee (for sales agents), Invoice (for sales transactions), and possibly others that link these two.\n\nIf you have access to such a database, you would need to query it to sum the total sales for each agent in 2009 and then identify the agent with the highest total. If you can provide more specific details or data, I can help guide you on how to perform this analysis.', additional_kwargs={'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 137, 'prompt_tokens': 51, 'total_tokens': 188, 'completion_tokens_details': {'audio_tokens': None, 'reasoning_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': None, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_159d8341cc', 'finish_reason': 'stop', 'logprobs': None}, id='run-99ea13b4-a339-41c8-8784-fa3f5586bc55-0', usage_metadata={'input_tokens': 51, 'output_tokens': 137, 'total_tokens': 188, 'input_token_details': {'cache_read': 0}, 'output_token_details': {'reasoning': 0}})

tool(msg)

# OUTPUT:
# "Error: table_names {'Human: Which sales agent made the most in sales in 2009?\\nAI: \\nTool: Album'} not found in database"


llm.bind_tools([tool]).invoke([msg])

# OUTPUT:
# AIMessage(content='', additional_kwargs={'tool_calls': [{'id': 'call_1svwD1QfqSnoKXcFoLNhMRlt', 'function': {'arguments': '{"table_names":"Employee, Invoice"}', 'name': 'sql_db_schema'}, 'type': 'function'}], 'refusal': None}, response_metadata={'token_usage': {'completion_tokens': 18, 'prompt_tokens': 124, 'total_tokens': 142, 'completion_tokens_details': {'audio_tokens': None, 'reasoning_tokens': 0}, 'prompt_tokens_details': {'audio_tokens': None, 'cached_tokens': 0}}, 'model_name': 'gpt-4o-2024-08-06', 'system_fingerprint': 'fp_45cf54deae', 'finish_reason': 'tool_calls', 'logprobs': None}, id='run-6a4ed529-2298-4841-b305-1fc6bd5629ef-0', tool_calls=[{'name': 'sql_db_schema', 'args': {'table_names': 'Employee, Invoice'}, 'id': 'call_1svwD1QfqSnoKXcFoLNhMRlt', 'type': 'tool_call'}], usage_metadata={'input_tokens': 124, 'output_tokens': 18, 'total_tokens': 142, 'input_token_details': {'cache_read': 0}, 'output_token_details': {'reasoning': 0}})

Description

I am using the following tutorial for making a SQL agent: https://langchain-ai.github.io/langgraph/tutorials/sql-agent/

In the tutorial, there is a step that returns only RELEVANT tables to answer the user query. It is supposed to look at all available/usable tables and pick which of them may be relevant to the query.

In the attached code to this thread, I reconstructed the call. The query I use for this was fetched from the original tutorial, and passes the state["messages"] to this step which is GPT4o llm call with bind_tools. It basicall uses "sql_db_schema" tool from SQLDatabaseToolkit.

I imagined that in the background there is some prompt being invoked, but when I recreated the steps from langchain's source it still works.

The description of "sql_db_schema" states "Get the schema and sample rows for the specified SQL tables.".

I am not adding any system prompt, nor initializing chains for tables or anything.

My confusion and question is: At what time does the combination of llm and tool decides to pick the relevant tables only? And more importantly, where and how?

I am kind of new to AI agents, so any help around this will be very appreciated.
Thanks!

System Info

langchain==0.3.4
langchain-community==0.3.3
langchain-core==0.3.13
langchain-openai==0.2.4
langchain-plantuml==0.0.7
langchain-text-splitters==0.3.0
langchainhub==0.1.21

Python 3.9.20