[help] Rerunning Experiments for Replicating Paper Results

[kevin3567]

For paper replication purposes, in the Readme, it is stated that for "Every metric was collected by running the experiment 10 times separately and calculating the average value." Is this done only for collecting the training/inference speed/gpu-usage, or is this also applicable for getting the reported task-specific (i.e. arc-e, boolq, etc.) accuracy scores?

[mikecovlee]

We collected these performance metrics by running the evaluation 10 separate times and calculating the average values. For accuracy scores, running multiple trials is unnecessary because our code ensures reproducibility when the random seed is fixed in the same environment and on the same device.