This repository was archived by the owner on Feb 15, 2025. It is now read-only.
feat:(backend): Implement Context Cancellation #280
Labels
enhancement
New feature or request
Milestone
Type: Feature
Description: Cancelling context cancels generation to free resources to retry requests or otherwise stop inference.
User Story:
For each backend (VLLM, Embeddings, Llama-cpp-python, Whisper), implement any changes from the SDK required to perform context cancellation.
Acceptance Criteria:
The text was updated successfully, but these errors were encountered: