Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not validate model requests and responses #2230

Open
jvstme opened this issue Jan 27, 2025 · 0 comments
Open

Do not validate model requests and responses #2230

jvstme opened this issue Jan 27, 2025 · 0 comments
Labels
enhancement A non-feature improvement

Comments

@jvstme
Copy link
Collaborator

jvstme commented Jan 27, 2025

Problem

Currently, dstack's OpenAI-compatible endpoint performs strict validation of all requests and responses against a hardcoded schema. However, some model launchers don't adhere to the official OpenAI schema, and the official schema can also evolve rapidly. As a result, some models or launchers can't be used with dstack because validation fails for them. Examples: #1777, #2228.

Solution

Do not perform strict validation for format: openai models. Only extract the fields that dstack needs, such as the model field, and pass the rest of the fields in requests and responses as is.

Note: strict validation may still be necessary for format: tgi models in order to perform format conversion.

@jvstme jvstme added the enhancement A non-feature improvement label Jan 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement A non-feature improvement
Projects
None yet
Development

No branches or pull requests

1 participant