Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FT] Add Gemba MQM Translation Metric #397

Open
JoelNiklaus opened this issue Nov 19, 2024 · 2 comments
Open

[FT] Add Gemba MQM Translation Metric #397

JoelNiklaus opened this issue Nov 19, 2024 · 2 comments
Labels
feature request New feature/request

Comments

@JoelNiklaus
Copy link
Contributor

Issue encountered

The metrics only include rather outdated translation metrics.

Solution/Feature

Gemba MQM seems to be a current metric. Adding it would make translation evaluation better.

@JoelNiklaus JoelNiklaus added the feature request New feature/request label Nov 19, 2024
@NathanHB
Copy link
Member

Looks nice ! This is using external APIs and does not seem to have a PyPI package so we would need to implement it in Lighteval. This not not high priority but if you need it feel free to open a PR and we can help you set it up :)

@JoelNiklaus
Copy link
Contributor Author

Great, thanks!
Yes, I see two avenues:

  1. Fork their repo, publish a pip package and integrate it like that.
  2. Just copy their prompts and post processing functions.

IMO option 1 is cleaner and also allows other people to use the metric more easily.

@chuandudx Would you be interested in taking this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature/request
Projects
None yet
Development

No branches or pull requests

2 participants