Continuous Performance Benchmarking

Qi uses continuous performance benchmarking as part of its CI workflows in order to track performance changes across commits. The results of benchmarking and trends over time are available at https://countvajhula.github.io/qi/benchmarks/

This is accomplished by using github-action-benchmark.

How It's Set Up

• There is a workflow file in .github/workflows/benchmarks.yml
• Like most development workflows, the CI workflow is configured to run a Makefile target, in this case make report-benchmarks, to generate the performance data. This can be run locally as well, and would produce the data in JSON format as standard output (STDOUT).
• In the benchmarks.yml workflow, the output from the Make target is dumped into a file called benchmarks.txt.
• github-action-benchmark is then invoked, and it expects the input as a JSON-formatted text file -- we simply tell it, via workflow config, that it can use the file benchmarks.txt that was produced by the previous step.
• This GitHub Action works by pushing the data and an HTML facade to the gh-pages branch of the Qi repo, at the specified folder (benchmarks).
• Note that the gh-pages branch is also being used to host backup docs via the GitHub Pages Deploy action, so we need to tell that workflow (docs.yml) to exclude the benchmarks folder from its "clean" step, so that it doesn't delete the benchmarks whenever there is an update to the documentation.
• The deployed HTML page, including the benchmark data in charts indexed by commit, are available at https://countvajhula.github.io/qi/benchmarks/

Manually Adding Data

Data is automatically tracked on commits to the main branch, and manually editing the data should generally be avoided. But there are a few cases where you might want to do this, for instance, if the data got corrupted and you'd like to fix it, or if there is a new data point you've added to the tracking, and would like to retroactively add the value prior to some new changes that you believe affected it. In such cases, the data should be generated using GitHub Actions directly, and not locally or via some form of estimation from local analysis, to ensure that it is accurate. Here are the steps to do this:

1. Create a new branch from the specific commit you'd like to benchmark
2. Add the necessary configuration you need in order to produce the value you are interested in
3. Modify the benchmarks.yml workflow file so that (1) it only generates the data but doesn't invoke the github-action-benchmark step -- i.e. remove that step, and (2) modify the on trigger condition so that it runs on any commit, not only on main branch commits.
4. Push your branch to GitHub, and find the generated data in the output for the "Run benchmark" step
5. Re-run the job as many times as you need, and compute the average or statistically reasonable value
6. Checkout the gh-pages branch locally
7. Manually modify the JSON file benchmarks/data.json to include the new data point(s)
8. Ensure the JSON passes a validation tool
9. Commit it and push it upstream
10. The new data should show up in a few minutes, after the "pages deploy" job has completed

See this PR for an example.