Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch token count issue while streaming with customized models #3241

Merged
merged 6 commits into from
Sep 25, 2024

Conversation

BeibinLi
Copy link
Collaborator

If llama, llava, phi, or some other models are used for streaming (with stream=True), the current design would crash after fetching the response.

A warning is enough in this case, just like the non-streaming use cases.

Why are these changes needed?

Related issue number

Checks

If llama, llava, phi, or some other models are used for streaming (with stream=True), the current design would crash after fetching the response.

A warning is enough in this case, just like the non-streaming use cases.
@codecov-commenter
Copy link

codecov-commenter commented Jul 28, 2024

Codecov Report

Attention: Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.

Project coverage is 21.29%. Comparing base (6aaa238) to head (7d1a110).
Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
autogen/oai/client.py 0.00% 5 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##             main    #3241       +/-   ##
===========================================
- Coverage   33.24%   21.29%   -11.95%     
===========================================
  Files          99       99               
  Lines       11016    11020        +4     
  Branches     2365     2537      +172     
===========================================
- Hits         3662     2347     -1315     
- Misses       7026     8507     +1481     
+ Partials      328      166      -162     
Flag Coverage Δ
unittests 21.26% <0.00%> (-11.99%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

autogen/oai/client.py Outdated Show resolved Hide resolved
@jackgerrits jackgerrits added this pull request to the merge queue Sep 25, 2024
Merged via the queue into main with commit ece6924 Sep 25, 2024
145 of 157 checks passed
@jackgerrits jackgerrits deleted the stream-token-count branch September 25, 2024 15:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants