Enable Code Carbon by default in evaluation runs

[isaac-chung]

[Edit] Was discussing with @KennethEnevoldsen about this suggestion as a way to combine run eval time + hardware info so we don't have to keep track of both. Enabling this would be a convenient step towards that before we decide to use something else that is more representative (such as FLOPS).

[Muennighoff]

What's the motivation for enabling it by default? I think @imenelydiaker intentionally made it optional in #712 (comment) 🤔

[isaac-chung]

Was discussing with @KennethEnevoldsen about this suggestion as a way to combine run eval time + hardware info so we don't have to keep track of both. Enabling this would be a convenient step towards that before we decide to use something else that is more representative (such as FLOPS).

[Muennighoff]

Makes sense, thanks for the context!!

[isaac-chung]

No worries, my bad for not filling out the description in the first place. Updated!

[isaac-chung]

FYI all: (copied from the PR)
Based one this issue, it seems like we can* allow multiple runs, but the tracking will be somewhat skewed. It won't matter in the tests, but will for sure affect real runs if multiple runs are done on the same machine. I think we would also add a line in README and/or a warning to let users know.

[Samoed]

CO² added to open llm leaderboard https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

[isaac-chung]

Merged into branch v2.0.0