Skip to content

Continuous Performance Benchmarking

Siddhartha Kasivajhula edited this page Jul 5, 2022 · 6 revisions

Qi uses github-action-benchmark to track performance changes across commits. Here's how it's set up:

  • There is a workflow file in .github/workflows/benchmarks.yml
  • Like most development workflows, the CI workflow is configured to run a Makefile target, in this case make report-benchmarks, to generate the performance data. This can be run locally as well, and would produce the data in JSON format as standard output (STDOUT).
  • In the benchmarks.yml workflow, the output from the Make target is dumped into a file called benchmarks.txt.
  • github-action-benchmark is then invoked, and it expects the input as a JSON-formatted text file -- we simply tell it, via workflow config, that it can use the file benchmarks.txt that was produced by the previous step.
  • This GitHub Action works by pushing the data and an HTML facade to the gh-pages branch of the Qi repo, at the specified folder (benchmarks).
  • Note that the gh-pages branch is also being used to host backup docs via the GitHub Pages Deploy action, so we need to tell that workflow (docs.yml) to exclude the benchmarks folder from its "clean" step, so that it doesn't delete the benchmarks whenever there is an update to the documentation.
  • The deployed HTML page, including the benchmark data in charts indexed by commit, are available at https://countvajhula.github.io/qi/benchmarks/
Clone this wiki locally