This repository was archived by the owner on Feb 15, 2025. It is now read-only.
Context Cancellation #139
Labels
bug
Something isn't working
enhancement
New feature or request
good first issue
Good for newcomers
help wanted
Extra attention is needed
Comments
gerred
added
good first issue
|
@edengebrezgi did you happen to start this ticket yet? If not, would you mind if I took over? |
|
@vanakema no body has started on this one yet! You can work on it if you'd like. Let me know if you have questions or want to pair on it! |
|
@justinthelaw would love to pair on it! How should we coordinate? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
cc @DayOfThePenguin
One issue right now is even if gRPC gets cancelled, backend models will continue generating into the void. This can cause model servers to seemingly hang while the thread is blocked.
We need some sort of async cancellation mechanism to stop a generation.
The text was updated successfully, but these errors were encountered: