Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Catch token count issue while streaming with customized models #3241

Merged
merged 6 commits into from
Sep 25, 2024
Merged

Catch token count issue while streaming with customized models #3241

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a40aa56
Catch token count issue while streaming with customized models
BeibinLi Jul 28, 2024
c27f0a9
Only catch not implemented error
BeibinLi Jul 29, 2024
beb105a
Merge branch 'main' into stream-token-count
BeibinLi Jul 29, 2024
286d647
Merge branch 'main' into stream-token-count
sonichi Aug 6, 2024
7d1a110
Merge branch 'main' into stream-token-count
jackgerrits Sep 25, 2024
86b9089
Merge branch 'main' into stream-token-count
jackgerrits Sep 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion autogen/oai/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -279,7 +279,12 @@ def create(self, params: Dict[str, Any]) -> ChatCompletion:

# Prepare the final ChatCompletion object based on the accumulated data
model = chunk.model.replace("gpt-35", "gpt-3.5") # hack for Azure API
prompt_tokens = count_token(params["messages"], model)
try:
prompt_tokens = count_token(params["messages"], model)
except NotImplementedError as e:
# Catch token calculation error if streaming with customized models.
logger.warning(str(e))
prompt_tokens = 0
jackgerrits marked this conversation as resolved.
Show resolved Hide resolved
response = ChatCompletion(
id=chunk.id,
model=chunk.model,
Expand Down
Loading