Add tokens count in TS SDK

[alexeygoncharov]

Do we have this counting system in here? https://colab.research.google.com/drive/1cY_YPaV0KUdveAj_nwy7V8gVEDZh_GDz#scrollTo=XBm0Dz-P-8Fm

[chitalian]

+1

[kakhulu31]

+1

[chitalian]

I dug a bit into this.

There seem to be a few approaches.

1. Mirror the python implementation: https://github.com/anthropics/anthropic-sdk-python/blob/main/anthropic/tokenizer.py#L51
   However this will require us to get tokenizer to compile to WASM
   JS / WebAssembly binding planned ? huggingface/tokenizers#63
   Support wasm huggingface/tokenizers#935

2. We can mirror what is happening here: https://github.com/niieani/gpt-tokenizer/blob/main/src/main.ts#L129-L144
   But I think this way is a bit more sketch.

Any other ideas for supporting this that is a bit easier or lower hanging?

[eek]

Maybe we can use https://github.com/dqbd/tiktoken since it's a BPE tokenizer as well and works great.

[niieani]

Author of gpt-tokenizer here. Just FYI I've rewritten gpt-tokenizer to be a complete TypeScript port of OpenAI's tiktoken (with some extra features sprinkled on top). Would be happy to accept a PR to include support for anthropics models.

[transitive-bullshit]

See also https://github.com/transitive-bullshit/compare-tokenizers

[bobber205]

This is extremely needed and important to determine cost/usage

[rattrayalex]

Good news – we expect have a separate package with this functionality released within a few days!

[bobber205]

@rattrayalex That's awesome! I know it's a big ask but do you mind commenting on this thread when it's available?

[rattrayalex]

Of course!

…

On Wed, Jun 28, 2023 at 2:27 PM, Alex Wait ***@***.***> wrote: @rattrayalex <https://github.com/rattrayalex> That's awesome! I know it's a big ask but do you mind commenting on this thread when it's available? — Reply to this email directly, view it on GitHub <#16 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFL6LSDFH5JKERI7ECA2UTXNRZSNANCNFSM6AAAAAAW734YU4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[rattrayalex]

This is now released! https://www.npmjs.com/package/@anthropic-ai/tokenizer

[bobber205]

@rattrayalex Does this help with completion token counts? That's the much more expensive one.

[rattrayalex]

Yes, you can use this to tokenize the responses as well.

[bobber205]

D'oh! Of course! Thanks!