Skip to content
This repository was archived by the owner on Feb 15, 2025. It is now read-only.

feat(sdk): context cancellation #279

Open
2 tasks
Tracked by #320
gphorvath opened this issue Mar 21, 2024 · 0 comments
Open
2 tasks
Tracked by #320

feat(sdk): context cancellation #279

gphorvath opened this issue Mar 21, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@gphorvath
Copy link

Type: Feature

Description: Cancelling context cancels generation to free resources to retry requests or otherwise stop inference.

User Story:
Changes to the SDK are required to broker context cancellation between the backends and API.

Acceptance Criteria:

  • Implement network cancellation hooks for backends in SDK
  • If necessary, modify GRPC as the mechanism for cancelling a request
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants