implement get_num_tokens to use google's count_tokens function #10565
Conversation
can get the correct token count instead of using gpt-2 model
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
dosubot
bot
added
Ɑ: models
|
@lkuligin @eyurtsev @hwchase17 |
|
@holtskinner can you help me review this PR, please? |
|
@baskaryan Can you help me review this? |
|
@lkuligin @eyurtsev @hwchase17 @holtskinner |
|
I don't think there's a need to add additional client and deal with it. You can use |
|
@lkuligin Please correct me if i am wrong. it looks like TextModel didn't have count_tokens function. >>> llm = VertexAI()
>>> "count_tokens" in dir(llm.client)
False
>>> llm.client.couont_tokens
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: '_PreviewTextGenerationModel' object has no attribute 'couont_tokens' |
|
ohh, this function is release by google-cloud-aiplatform v1.34.0 version, which release on Oct6 |
|
@lkuligin I did some modifications base on your suggestions. Please help us review it again when you are available. Thank you. :) |
|
Hi @lkuligin : |
|
@lkuligin Sorry to bother you. Can you review this change? please |
|
@lkuligin @eyurtsev @hwchase17 @holtskinner |
|
I can't approve/merge this PR, but LGTM. |
| @@ -97,3 +100,26 @@ async def test_model_garden_agenerate() -> None: | |||
| output = await llm.agenerate(["What is the meaning of life?", "How much is 2+2"]) | |||
| assert isinstance(output, LLMResult) | |||
| assert len(output.generations) == 2 | |||
|
|
|||
|
|
|||
| def test_vertex_call_trigger_count_tokens(mocker) -> None: | |||
There was a problem hiding this comment.
nits: I'm not sure this qualifies as an integration test. Probably, for a integration test we should rather actually invoke a call to VertexAI and check that it returns results successfully, wdyt?
There was a problem hiding this comment.
Hi @lkuligin:
For this testcase (test_vertex_call_trigger_count_tokens), i modify it, so it will call the underlying function directory, I also create the other testcase called "test_get_num_tokens_be_called_when_using_mapreduce_chain", because the token usage didn't be included from chain output. So it is hard to me to make sure if this function(count_tokens) will be called when user are using mapreduce method or not. This is why i decide to use marker to test this logic.
Please let me know your thoughts, or any suggestions i can make it more better. :)
…feature/enhance-vertexai-llm
baskaryan
added
the
lgtm
…hain-ai#10565) can get the correct token count instead of using gpt-2 model **Description:** Implement get_num_tokens within VertexLLM to use google's count_tokens function. (https://cloud.google.com/vertex-ai/docs/generative-ai/get-token-count). So we don't need to download gpt-2 model from huggingface, also when we do the mapreduce chain we can get correct token count. **Tag maintainer:** @lkuligin **Twitter handle:** My twitter: @abehsu1992626 --------- Co-authored-by: Bagatur <[email protected]>
…hain-ai#10565) can get the correct token count instead of using gpt-2 model **Description:** Implement get_num_tokens within VertexLLM to use google's count_tokens function. (https://cloud.google.com/vertex-ai/docs/generative-ai/get-token-count). So we don't need to download gpt-2 model from huggingface, also when we do the mapreduce chain we can get correct token count. **Tag maintainer:** @lkuligin **Twitter handle:** My twitter: @abehsu1992626 --------- Co-authored-by: Bagatur <[email protected]>
can get the correct token count instead of using gpt-2 model
Description:
Implement get_num_tokens within VertexLLM to use google's count_tokens function. (https://cloud.google.com/vertex-ai/docs/generative-ai/get-token-count). So we don't need to download gpt-2 model from huggingface, also when we do the mapreduce chain we can get correct token count.
Tag maintainer:
@lkuligin
Twitter handle:
My twitter: @abehsu1992626