Using Model from Ollama in ChatOpenAI doesnt invoke the tools with bind_tools

[AtmehEsraa]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3:8b-instruct-fp16",
    base_url="http://localhost:11434/v1",
)

Description

Using Model from Ollama in ChatOpenAI doesnt invoke the tools with bind_tools

System Info

..

[dosubot[bot]]

To address the issue of invoking tools with bind_tools when using the Ollama model in ChatOpenAI, ensure you're correctly binding your tools to the chat model. Here's a concise guide:

1. Bind Tools Correctly: Use the bind_tools method to attach your tools to the ChatOpenAI instance. This involves creating tool instances and converting them into a format that ChatOpenAI can use. For example:

from langchain_openai import ChatOpenAI
from langchain.tools import MoveFileTool, format_tool_to_openai_function

llm = ChatOpenAI(
    api_key="ollama",
    model="llama3:8b-instruct-fp16",
    base_url="http://localhost:11434/v1",
)

tools = [MoveFileTool()]
functions = [format_tool_to_openai_function(t) for t in tools]

llm_with_tools = llm.bind_tools(tools=functions)

2. Check Tool Compatibility: Ensure your tools are compatible with ChatOpenAI. Tools can be dictionaries, Pydantic models, callable functions, or BaseTool instances.

3. Error Handling: If you encounter errors, verify that you're not passing unsupported arguments to bind_tools and that your tools are correctly defined and formatted.

4. Structured Output and Schema: When using structured output, ensure a valid schema is specified. This can be a Pydantic class or a dictionary.

If following these steps doesn't resolve your issue, please provide more details about the errors or behavior you're experiencing for further assistance.

Details

Sources

• AttributeError: 'ChatGoogleGenerativeAI' object has no attribute 'bind_tools'
• Functions not called when i use langchain_openai.ChatOpenAI as llm
• Tools within ChatModel
• Issue with Custom Tools Invocation and Auto-generated Responses from Langchain LLM
• ChatOpenAI with "bind_tools", If use "base_url" other API sever, dont call "tool" and don't response "tool_calls"
• libs/partners/openai/langchain_openai/chat_models/base.py
• libs/experimental/langchain_experimental/llms/ollama_functions.py
• libs/partners/openai/tests/integration_tests/chat_models/test_base.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[maximeperrindev]

Hi @AtmehEsraa,

You cannot use ChatOpenAI with an Ollama model.
Moreover, Ollama chat model does not support tool calling : https://python.langchain.com/v0.1/docs/integrations/chat/

[AtmehEsraa]

@maximeperrindev so how we can solve this issue ?

[maximeperrindev]

@AtmehEsraa,

Well, if you want to use Ollama you can't. The workaround will be to use one of the models supporting tool calling.

[romilandc]

@maximeperrindev That's incorrect. You CAN use ChatOpenAI with Ollama you must pull model via Ollama first however. OP's code from langchain_openai import ChatOpenAI llm = ChatOpenAI( api_key="ollama", model="llama3:8b-instruct-fp16", base_url="http://localhost:11434/v1", ) should work as long as he has the an Ollama instance active when using ChatOpenAI().

[eyurtsev]

See this guide to add ad-hoc too calling capability to models that do not support it natively: https://python.langchain.com/v0.2/docs/how_to/tools_prompting/

Keep in mind that the gap in quality can be very big between a model that's been fine-tuned for tool calling and one that hasn't.

For tool calling models see:

https://python.langchain.com/docs/how_to/tool_calling/

And a list of such models here: https://python.langchain.com/docs/integrations/chat/

[cboettig]

Thanks for clarifying this @eyurtsev , super helpful. from those docs:

Tool calling is not universal, but many popular LLM providers, including Anthropic, Cohere, Google, Mistral, OpenAI, and others, support variants of a tool calling feature.

Apologies if this is a naiive question, but do any of the models listed there have support for self-hosting the llm in tool-calling mode?

That is, I know that we can download and run the open model weights for models from Cohere and Mistral via ollama and other mechanisms. I also see ollama-langchain explicitly does not support tooling, though that feels a bit apples-to-oranges as ollama obviously isn't itself a model but only an interface to collection of models, some of which are and some of which are not tuned for tools.

The examples with tool chains for Cohere and Mistral show using the corresponding API keys to access the hosted versions of those models. But I cannot tell if that is the only option for using these models in tool form (i.e. that they cannot be self-hosted in this mode, at least not within existing langchain tooling), or merely because that is the most convenient. Thanks!

[eyurtsev]

Hi @cboettig ,

Not a naive question at all. I don't know which features are supported now by ollama. The langchain integration with ollama definitely does not support tools currently.

According to these docs: https://github.com/ollama/ollama/blob/main/docs/openai.md, JSON mode exists, but not function calling, but this is using the openai format.

If you find more, please share!

[Atul227]

Have any body tried tool calling using meta--llama3.1-70b-instruct with ChatOpenAI as soon as i switch from gpt-4 to meta--llama3.1-70b-instruct it doesn't perform tool call

[romilandc]

Getting similar error when using llm.with_structured_output() within a function chained with llm.bind_tools(tools).

[yaoduo11]

Is there any good solution?

[ssoto]

Same here, it never calls to the functions.

Should we use OpenAI API or go to OllamaFunctions?

[lemassykoi]

I used instructions from https://python.langchain.com/v0.2/docs/tutorials/qa_chat_history/#tying-it-together-1
I'm not using bind_tools, but I'm using ChatOpenAI instead of ChatOllama, to create my retriever

This is working for me at the moment:

import bs4
from langchain_chroma import Chroma
from langchain_community.document_loaders import WebBaseLoader
from langchain_community.embeddings import OllamaEmbeddings
#from langchain_community.chat_models import ChatOllama
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import ChatOpenAI
from langchain.tools.retriever import create_retriever_tool
from langchain_core.messages import HumanMessage
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.sqlite import SqliteSaver

current_model_name  = "llama3-custom"
current_embed_model = "snowflake-arctic-embed:latest"
ollama_base_url     = 'http://127.0.0.1:11434'
memory              = SqliteSaver.from_conn_string(":memory:")

#llm = ChatOllama(model=current_model_name, num_ctx=8192)
llm = ChatOpenAI(model=current_model_name, api_key='ollama', base_url='http://127.0.0.1:11434/v1', temperature=0)

### Construct retriever ###
loader = WebBaseLoader(
    web_paths=("https://lilianweng.github.io/posts/2023-06-23-agent/",),
    bs_kwargs=dict(
        parse_only=bs4.SoupStrainer(
            class_=("post-content", "post-title", "post-header")
        )
    ),
)
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
splits = text_splitter.split_documents(docs)
vectorstore = Chroma.from_documents(documents=splits, embedding=OllamaEmbeddings(base_url=ollama_base_url, model=current_embed_model, show_progress=True,))
retriever = vectorstore.as_retriever()

tool = create_retriever_tool(
    retriever,
    "blog_post_retriever",
    "Searches and returns excerpts from the Autonomous Agents blog post.",
)
tools = [tool]
agent_executor = create_react_agent(llm, tools, checkpointer=memory, debug=True)


config = {"configurable": {"thread_id": "TEST005"}}

# Method 1 - test tool without llm
print('\n==> Retriever Tool')
print(tool.invoke("task decomposition"))


## Method 2
print('\n==> Retriever Tool Step 2')
for s in agent_executor.stream({"messages": [HumanMessage(content="Hi! I'm Bob")]}, config=config):
    print(s)
    print("----")

print('\n==> Retriever Tool Step 3')
query = "What is Task Decomposition?"
for s in agent_executor.stream({"messages": [HumanMessage(content=query)]}, config=config):
    print(s)
    print("----")

[x51xxx]

You can use it like this https://gist.github.com/x51xxx/4d61e8c675681d165f012a7231d06976
In Python, you can use an analogy.

[snailUlitka]

Now Ollama with llama3.1 can use tool calling (source), but llama3.1:8b is pretty bad at it. I use bind_tools with tool_choice, where I specify the name of the tool, but llama ignores this and tries to answer only in the content field

[cboettig]

@snailUlitka right, try without wrapping ChatOpenAI. You can do tool calling with ChatOllama directly, e.g. see here. It does call tools, but does not always call the right tool or all the tools. groq is considerably better at this, though I think not as good as the OpenAI and Anthropic models.

[snailUlitka]

@cboettig I apologize, I still used ChatOllama from langchain_community and there bind_tools doesn't implemented. I didn't notice the moment of change, and I didn't see any messages about deprecated module. Thank you for your help and hint!

[whatwehaveunlearned]

I’ve been working on integrating Ollama with LangChain tools. Based on various posts, I’ve seen several approaches that seem to work, but are becoming obsolete due to the use of initialize_agent.

I understand LangChain is transitioning towards using .bind to attach tools to LLMs and employing the create_tool_calling_agent constructor instead. However, I was under the impression that this functionality hadn’t been fully implemented yet—until I read your recent posts.

However, @cboettig, when I reviewed the code in the link you provided, which aligns with the structure I expected for the new release, I encountered the same issue I was running into. The .bind method isn’t implemented for the LLM. I thought using the model llama3-groq-tool-use would solve this, but it hasn’t.

Is this approach working for anyone else, or is it no longer functional? Am I missing something?

It seems like outdated code, especially since even the import statements appear incorrect; for example, from langchain_ollama import ChatOllama should now be from langchain_community.chat_models import ChatOllama

llm = ChatOllama(
    model="llama3-groq-tool-use:70b",
   # model="llama3.1",
    temperature=0,
)
agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({
    "system": "You are a helpful agent.  Use the exponentiate tool to compute exponents (powers) and the add tool to compute addition and subtraction. Always try to determine and invoke a tool to perform your tasks.  If you do not have enough information, ask for more details.",
    "input": "what's 3 plus the value of 5 raised to the 2.743. Then substract 918.1241", })

[snailUlitka]

@whatwehaveunlearned Maybe I didn't quite understand the problem. I use LangGraph myself, but I've previously used other ways to create LLM agents.

It seems to me that now the easiest way to use tool calling is to call LLM bind_tools. In the case of Ollama, it is important to use import from partners, e. g. from langchain_ollama.chat_models import ChatOllama. Because with langchain_community.chat_models.ollama i getting NotImplementedError

bind_tools is bind with convert_to_openai_tool method which converts from LangChain tool to JSON object.

Simple example:

from langchain_ollama.chat_models import ChatOllama
from langchain_core.tools import tool

@tool
def get_weather(city: str):
    """Give me name of city, I return current weather"""
    return "19 degree and foggy"

llm = ChatOllama(model="llama3-groq-tool-use:8b").bind_tools(tools=[get_weather])

res = llm.invoke("what is the weather in London?")

res.tool_calls[0]
>>> {
...    'name': 'get_weather',
...    'args': {'city': 'London'},
...    'id': '95ab0727-adc7-44d5-9fbc-50ca5aa3510d',
...    'type': 'tool_call'
... }

[whatwehaveunlearned]

I see. It works with lanchain_ollama import but not with the langchain_community import. Does anyone know why this is the case?

[dosubot[bot]]

I found some similar discussions and issues that might help you:

1. bind_tools NotImplementedError when using ChatOllama: This issue is open and discusses the NotImplementedError when using bind_tools with ChatOllama. The suggested solution is to use Ollama's OpenAI compatible API ^[1].

2. bind_tools is not available in Ollama: This discussion highlights the absence of bind_tools in Ollama and suggests using the OpenAI compatible API ^[2].

3. LangChain is incorrectly parsing OpenAI tool calls as valid: This discussion addresses issues with tool call parsing and provides a solution to enhance the default_tool_parser function for better validation ^[3].

To use the model from Ollama in ChatOpenAI to invoke tools with bind_tools, follow these steps:

1. Install the langchain-ollama package:

   pip install -U langchain-ollama

2. Instantiate the ChatOllama class:

   from langchain_ollama import ChatOllama
   
   llm = ChatOllama(model="llama3.1", temperature=0)

3. Define the tools using Pydantic classes, dict schemas, LangChain tools, or functions:

   from pydantic import BaseModel, Field
   
   class GetWeather(BaseModel):
       """Get the current weather in a given location"""
       location: str = Field(..., description="The city and state, e.g. San Francisco, CA")

4. Bind the tools to the model:

   llm_with_tools = llm.bind_tools([GetWeather])

5. Invoke the model with a message:

   ai_msg = llm_with_tools.invoke("what is the weather like in San Francisco")

6. Access the tool calls from the AIMessage:

   ai_msg.tool_calls

Here is the complete code snippet:

# Step 1: Install the package
pip install -U langchain-ollama

# Step 2: Instantiate the ChatOllama class
from langchain_ollama import ChatOllama

llm = ChatOllama(model="llama3.1", temperature=0)

# Step 3: Define the tools
from pydantic import BaseModel, Field

class GetWeather(BaseModel):
    """Get the current weather in a given location"""
    location: str = Field(..., description="The city and state, e.g. San Francisco, CA")

# Step 4: Bind the tools to the model
llm_with_tools = llm.bind_tools([GetWeather])

# Step 5: Invoke the model with a message
ai_msg = llm_with_tools.invoke("what is the weather like in San Francisco")

# Step 6: Access the tool calls
ai_msg.tool_calls

This should allow you to use the model from Ollama in ChatOpenAI to invoke tools with bind_tools ^[4][5][6][7].

To continue talking to Dosu, mention @dosu.