Token counting for Anthropic using the values in the API response

[sihil]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

OpenAI has a callback handler which provides the number of tokens consumed as reported by the API. The Anthropic API has now started reporting similar usage numbers on the API response. It would be really useful to have this implemented for Anthropic.

The following fragment is included in Anthropic responses:

  "usage": {
    "input_tokens": 10,
    "output_tokens": 25
  }

Motivation

With the advent of Anthropic's multimodal models in Claude 3, this has become more important. The current approach of token counting on the client side no longer works.

Proposal (If applicable)

No response