Env Vars part of RunId, and Support for Machine-Specific Settings - 2025-02-03

This release has three major changes that will affect how ReBench interprets configurations,
and a change for denoise that requires an update of the sudoers file.

1. "Run" identity is now based on all details, e.g. also env vars (#271)

This change can affect the number of different runs ReBench identifies.
Until now, the identity of a "run" was based on the command line generated from the
configuration. However, this meant, we could not distinguish between runs
with for instance different environment variables.
From now on, the identity of runs is based on all details of the configuration,
which may mean that configuration files may result in more distinct runs
than they did previously.

2. Build commands are combined and run as a single shell script (#269)

The build: key takes a list of commands, which are now combined into a single
shell script. Previously, each command was run separately.
Since we avoid executing the same command multiple times,
treating commands separately led to confusing results and was not very useful.
By combining the commands into a single script, identifying redundant script
execution leads to more predictable results.

3. Support for machine-specific settings (#272)

A new lowest-priority configuration level is introduced, which is meant to define
machine-specific settings. This can be useful, for instance for settings like this:

machines:
  smaller-machine:
    cores: [1, 2]
  larger-machine:
    cores: [1, 2, 4, 8]

These settings are combined into a configuration based on the usual rules.
So, if a benchmark suite or a benchmark give specific values for cores:, these
have higher priority. However, one would likely use it, avoiding
benchmark-specific values.

When executing ReBench on the corresponding machine, these settings can be
selected with the -m option. Thus, the larger machine configuration is used
when running:

rebench -m larger-machine rebench.conf

4. rebench uses denoise.py directly as script (#281)

Previously, we used the rebench-denoise script that is automatically created
during the setup process. However, this required a setup where ReBench was
installed so that the root user had access to it with a proper PYTHONPATH.
To avoid issues with setup, avoid having to pass PYTHONPATH through sudo,
and to make the use of denoise more robust, we changed denoise.py so that it
can be used directly as a script by ReBench.

In practice, the means, it's likely that the sudoers file will need to be
updated to point to the denoise.py script directly. ReBench will output the
path to the script when it is not able to use it directly.

Other Minor Changes

Features

• add support for ReBenchDB API Version 2 (#236)
• expand ~ in paths, right before invocation (#240, #283)
• add summary for columns where all values are the same (#256)
• show env in debug output (#280)

Changes

• make output for currently running benchmark more compact (#237)
• invoke denoise with absolute path to avoid need for having it on a secure_path (#238, #273)
• rename the machines variable/filters to a tags variable/filter (#264)
• support Python 3.13 as latest version, drop support for Python 3.8 (#268)
• reduce number of git invocations to get source information (#275)
• distinguish rebench return codes for different errors (#282)

Bug Fixes

• added missing machine column heading for summary table (#246)
• behave more gracefully on bare-bone setup, e.g. without git (#245)
• raise error in TimeAdapter to match other adapters (#254)
• make profiling with perf more robust (#255)
• make denoise more robust to absent tools and running as root with a user-level installation (#260)
• fix handling of ctrl-c interrupts (#262)
• avoid unnecessary warning about data reporting, when no data is to be reported (#277)

Development

• use black formatting (#267)
• use mypy type checking and add some annotations (#270)
• use pip install —editable in CI to avoid incorrect coverage reporting (#279)
• add testing on Rocky Linux with integration test (#282)

Thanks to @antonzhukovin, @vext01, and @martinmcclure for their contributions!

Full Changelog: v1.2.0...v1.3.0