Skip to content

Improve error handling for invalid eval results in model cards #3000

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
Apr 16, 2025

Conversation

hanouticelina
Copy link
Contributor

When listing models with list_models(cardData=True), some models have incorrectly formatted eval results which causes errors during iteration:

[364]if self.model_name is None:
--> [365]   raise ValueError("Passing `eval_results` requires `model_name` to be set.")
ValueError: Passing `eval_results` requires `model_name` to be set.

This is flagged in #2894 and in this internal slack message.

This PR does two things:

  • Adds type validation for eval_results to ensure they're EvalResult objects.
  • when ignore_metadata_errors is true, allows users to ignore metadata errors (this is the case when parsing card data in ModelInfo).

@hanouticelina hanouticelina requested a review from Wauplin April 14, 2025 13:54
Copy link
Member

@julien-c julien-c left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what's weird is that this is already validated on the server side IIRC

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@hanouticelina
Copy link
Contributor Author

hanouticelina commented Apr 14, 2025

what's weird is that this is already validated on the server side IIRC

@julien-c it seems it's not the case, because here for example : https://huggingface.co/nozomuteruyo14/Diff_LoRA/blob/main/README.md, eval_results is a string

EDIT: doesn't exist server-side, but users can add a eval_results field manually in the metadata

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggesting a change so that we definitely get rid of these messy errors 😬

Copy link
Contributor

@Wauplin Wauplin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All good! Thanks for iterating :)

@hanouticelina hanouticelina merged commit 2cef17b into main Apr 16, 2025
24 checks passed
@hanouticelina hanouticelina deleted the eval-results-parsing branch April 16, 2025 12:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants