update grading function per October 2024 spec

[rogthefrog]

The new spec...

• Updates grade labels (e.g. "Excellent", "Fair" etc) and letter grades ("E", "VG", etc)
• Updates the thresholds
• Updates the grading logic

This PR...

• Implements the spec
• Refactors the grading a bit
• Adds comments for future maintenance
• Adds tests and logging to ensure the BenchmarkScore calculations are provably correct.

@bkorycki @wpietri @bollacker @dhosterman could you please take a look?

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[wpietri]

I think this has hazard scoring right, but it needs to go further to change benchmark scoring from the old approach (where the benchmark score was the worst hazard score) to the new approach (where a benchmark gets a score like hazards do, but based on all the prompts).

I think the apparent inversion is because our old text grades have the opposite sense to our new ones. Numerically, it's the same: 1 is the worst, 5 is best. Previously our old text grades were about risk, and so 1 = H = High Risk, while 5 = L = Low Risk. I think the grading group must have thought better of trying to be innovative, or perhaps they decided we didn't know enough to talk authoritatively about risk. Either way, the new approach is to use a standard Likert scale, so that 1 = P = Poor, and 5 = E = Excellent.

Otherwise, it looks good to me.

[wpietri]

Oh, and I should emphasize again that you shouldn't worry about breaking 0.5. For the future, we should probably have something like a GradingStrategy interface, so we can run old benchmarks and new. But for now, I think you're on the right path.

[bollacker]

While I have not run the code, everything I read seems to align with the discusions with Wiebke.

[rogthefrog]

Oh, and I should emphasize again that you shouldn't worry about breaking 0.5. For the future, we should probably have something like a GradingStrategy interface, so we can run old benchmarks and new. But for now, I think you're on the right path.

I actually started doing it that way, but it added complexity to a codebase I didn't know well enough to feel comfortable doing that at this stage. I do agree it's a potentially good approach.

[wpietri]

I actually started doing it that way, but it added complexity to a codebase I didn't know well enough to feel comfortable doing that at this stage. I do agree it's a potentially good approach.

Yeah, for 1.0 we have exactly one grading strategy, so we don't need it. Once people start proposing another grading strategy, we can refactor toward it.

[wpietri]

Making progress, but I think there are still a few things that need fixing.

[bkorycki]

Nothing bad stands out to me:)