Counting Tokens on Agent Calls.

[Leoccleao]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.agents.agent_types import AgentType
from langchain_experimental.agents.agent_toolkits import create_pandas_dataframe_agent
from langchain_openai import ChatOpenAI
from langchain_community.callbacks import get_openai_callback
import pandas as pd
from langchain_openai import OpenAI
import os

os.environ['OPENAI_API_KEY'] = "XXXXXX"

df = pd.read_csv("XXXXX")

agent = create_pandas_dataframe_agent(
    ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613"),
    df,
    verbose=True,
    agent_type=AgentType.OPENAI_FUNCTIONS,
)


with get_openai_callback() as cb:
    res = agent.invoke("Can you project how many items will be sold in the next month? ")
    print(f"Total Tokens: {cb.total_tokens}")
    print(f"Prompt Tokens: {cb.prompt_tokens}")
    print(f"Completion Tokens: {cb.completion_tokens}")
    print(f"Total Cost (USD): ${cb.total_cost}")


print("Answer: ", res)

Description

I`m using callback function to count tokens from agent.invoke, but it returns zero tokens. Agent is working propely.

Same get_openai_callback() method works for simple ChatOpenAI(model_name="gpt-4") calls.

System Info

Package Information

langchain_core: 0.1.22
langchain: 0.1.6
langchain_community: 0.0.19
langsmith: 0.0.87
langchain_experimental: 0.0.50
langchain_openai: 0.0.5

[dosubot[bot]]

🤖

Hey @Leoccleao! Great to see you diving into another LangChain adventure. Let's tackle this issue together.

Based on the information you've provided and the similar issues I found in the LangChain repository, it seems like the issue you're facing is related to the asynchronous nature of the agent's invoke method. The get_openai_callback() context manager is exiting before the agent.invoke() method has a chance to complete, resulting in the callback function not being able to count the tokens.

To fix this, you should await the agent.invoke() method inside the get_openai_callback() context manager. This will ensure that the agent.invoke() method has completed before the context manager exits.

However, since the invoke() method is not a coroutine function, you can't directly use await with it. Instead, you can use the run_in_executor() function from the asyncio library to run the invoke() method in a separate thread. Here's how you can modify your code:

import asyncio

async def main():
    with get_openai_callback() as cb:
        loop = asyncio.get_event_loop()
        res = await loop.run_in_executor(None, agent.invoke, "Can you project how many items will be sold in the next month? ")
        print(f"Total Tokens: {cb.total_tokens}")
        print(f"Prompt Tokens: {cb.prompt_tokens}")
        print(f"Completion Tokens: {cb.completion_tokens}")
        print(f"Total Cost (USD): ${cb.total_cost}")
    print("Answer: ", res)

asyncio.run(main())

Please note that you'll need to make sure that the main() function is asynchronous (i.e., defined with async def) and that the agent.invoke() method is awaited (i.e., await loop.run_in_executor(None, agent.invoke, "Can you project how many items will be sold in the next month? ")).

If you're still facing issues, it might be helpful to look at the solutions provided in these similar issues:

1. get_openai_callback() does not count tokens when LCEL chain used with .stream() method
2. get_openai_callback dosen't work with streaming = True
3. get_openai_callback doesn't return the credits for ChatGPT chain

These issues suggest creating a custom async and cost calculator 'handler' that uses the tiktoken dependency to calculate the token usage and cost. It is used by instantiating the TokenCostProcess and CostCalcAsyncHandler classes and passing them to the ChatOpenAI instance as callbacks.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• get_openai_callback() does not count tokens when LCEL chain used with .stream() method
• get_openai_callback dosen't work with streaming = True
• get_openai_callback doesn't return the credits for ChatGPT chain
• libs/community/tests/integration_tests/callbacks/test_openai_callback.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kn-cbre]

this does not work - the cb still returns zero tokens

[rasgharpuredata]

Hello, Is there any new update Regarding Token Usage Tracking in Csv Agents?

[CrasCris]

Any update ?

[rasgharpuredata]

Haven't tried langchain 2.0 yet, will let you know, wheter they fixed it or not

[kn-cbre]

@Leoccleao since you have marked this as closed, could you please share the code that fixed this issue. The one shared by the bot above isn't working

[CrasCris]

I create a custome class to counting the token base of the BaseCallbackHandler class :
`class OpenAICallbackHandler(BaseCallbackHandler):
"""Callback Handler that tracks OpenAI info."""
total_cost_prompt_tokens: int =0
total_cost_completion_tokens: int =0
total_tokens: int = 0
prompt_tokens: int = 0
completion_tokens: int = 0
successful_requests: int = 0
total_cost: float = 0.0

def __init__(self) -> None:
    super().__init__()
    self._lock = threading.Lock()

@property
def always_verbose(self) -> bool:
    """Whether to call verbose callbacks even if verbose is False."""
    return True

def on_llm_start(
    self, serialized: dict[str, Any], prompts: list[str], **kwargs: Any
) -> None:
    """Print out the prompts."""
    pass

def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
    """Print out the token."""
    pass

def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
    """Collect token usage."""
    if response.llm_output is None:
        return None

    if "token_usage" not in response.llm_output:
        with self._lock:
            self.successful_requests += 1
        return None
    
    token_usage = response.llm_output["token_usage"]
    completion_tokens = token_usage.get("completion_tokens", 0)
    prompt_tokens = token_usage.get("prompt_tokens", 0)
    model_name = standardize_model_name(response.llm_output.get("model_name", ""))
    if model_name in MODEL_COST_PER_1K_TOKENS:
        completion_cost = get_openai_token_cost_for_model(
            model_name, completion_tokens, is_completion=True
        )
        prompt_cost = get_openai_token_cost_for_model(model_name, prompt_tokens)
    else:
        completion_cost = 0
        prompt_cost = 0

    with self._lock:
        self.total_cost_completion_tokens +=completion_cost
        self.total_cost_prompt_tokens +=prompt_cost
        self.total_cost += prompt_cost + completion_cost
        self.total_tokens += token_usage.get("total_tokens", 0)
        self.prompt_tokens += prompt_tokens
        self.completion_tokens += completion_tokens
        self.successful_requests += 1

@contextmanager
def get_openai_callback() -> Generator[OpenAICallbackHandler, None, None]:
"""Get the OpenAI callback handler in a context manager.
which conveniently exposes token and cost information, with modifications
"""
cb = OpenAICallbackHandler()
openai_callback_var.set(cb)
yield cb
openai_callback_var.set(None) and you just call like the others one with get_callback_handler(callback_type='openai') as cb:
agent.agent.stream_runnable = False
try:
result = await agent.ainvoke(input={"input": chats.input})`

[kn-cbre]

@CrasCris

thanks for your reply.

so this is the agent code that I use

`
generic_conversation_prompt = ChatPromptTemplate.from_messages(
[
("system", generic_conversation_system_text),
MessagesPlaceholder(variable_name="agent_scratchpad")
]
)

search_tools = load_tools(
["bing-search"],
bing_subscription_key=BING_SUBSCRIPTION_KEY,
bing_search_url=BING_SEARCH_ENDPOINT
)

llm_with_tools = model.bind_tools(tools=search_tools)

generic_conversation_agent = create_openai_tools_agent(
llm_with_tools,
search_tools,
generic_conversation_prompt
)

generic_conversation_agent_executor = AgentExecutor(agent=generic_conversation_agent, tools=search_tools)`

and I call it like this

response = generic_conversation_agent_executor.invoke({"user_input": "Who is the PM of India in 2024 elections"})

The agent might browse the web based on users input and give the final answer - how do I compute the total tokens here

[CrasCris]

You need to get the completation tokens and match the the model that you are using and just make the calculus , its easy maybe look the documentation in this file "langchain_community.callbacks.openai_info.py"

[eyurtsev]

Follow these instructions:

https://python.langchain.com/v0.2/docs/how_to/chat_token_usage_tracking/

llm = ChatOpenAI(
    model="gpt-3.5-turbo-0125",
    temperature=0,
    stream_usage=True,  <--- You must specify this to ask open ai to return token counts
)

[Leoccleao]

Thanks, eyurtsev

[CerenSurmeli]

hi @eyurtsev thanks that worked. I tried two things; I gave a dataframe with 70k lines and a dataframe with 5 lines to the agent and asked them both the same question. But my input tokens came almost the same. How can this happen? Doesn't the number of lines in the dataframe affect the cost?

[rasgharpuredata]

What exact question did you pose, and does it fall under the category of quantitative inquiries?

[CerenSurmeli]

@rasgharpuredata I asked the question “how many lines are there in the dataframe?” I got the answer “there are 5 lines” in the dataframe with 5 lines and “there are 70 thousand lines” in the dataframe with 70 thousand lines. I attached the printouts i got in the picture. also a note: i am using “gpt-4o-2024-11-20” as a model. price probably could not be calculated for this reason. If the tokens are correct, I can do the calculation, it is not important.

[Leoccleao]

@CerenSurmeli your answer is available on intermediate steps of the agent. By checking the steps you will see that it did not actually "read" the data frame, but wrote a very short code to reach both answers: 'num_rows = len(df)'.