Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve async processing of chat messages #18

Open
rchan26 opened this issue Apr 18, 2024 · 0 comments
Open

Improve async processing of chat messages #18

rchan26 opened this issue Apr 18, 2024 · 0 comments

Comments

@rchan26
Copy link
Collaborator

rchan26 commented Apr 18, 2024

The current way the pipeline is implemented to ensure that we stay within a given QPM rate limit is that we wait before asynchronously sending another prompt. However, for chat instances, a prompt can be a list of messages that we send. This means we're actually waiting a time before initialising a chat and within a particular chat, we don't do any waiting between messages.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant