Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set up simple intersectional disaggregated evaluation through Evaluate #7

Open
NimaBoscarino opened this issue Nov 14, 2022 · 1 comment
Open

Set up simple intersectional disaggregated evaluation through Evaluate #7

NimaBoscarino opened this issue Nov 14, 2022 · 1 comment
Assignees
@NimaBoscarino
Labels
enhancement New feature or request
Milestone
v0.1.3

Comments

@NimaBoscarino
Copy link
Contributor

NimaBoscarino commented Nov 14, 2022 •
edited
Loading

Required:

  • A sample script showing how to take a HF dataset, disaggregate it, run an Evaluator on it, and display the resulting evaluation metrics disaggregated by a combination of disaggregation modules.
  • Ideally this should be built without having to fork Evaluate. If a custom Evaluator is required, it can be added to this repo and the evaluate dependency can be added as an extra (optional) dependency to this library.
@NimaBoscarino NimaBoscarino self-assigned this Nov 14, 2022
@NimaBoscarino NimaBoscarino added this to the v0.1.2 milestone Nov 14, 2022
@NimaBoscarino NimaBoscarino modified the milestones: v0.1.2, v0.1.3 Dec 5, 2022
@NimaBoscarino
Copy link
Contributor Author

Moving to v0.1.3.

@NimaBoscarino NimaBoscarino added the enhancement New feature or request label Dec 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@NimaBoscarino NimaBoscarino

Labels
enhancement New feature or request
Projects
None yet
Milestone
v0.1.3
Development

When branches are created from issues, their pull requests are automatically linked.

7/intersectional-evaluate huggingface/disaggregators
1 participant
@NimaBoscarino