Benchmarking

In order to achieve a certain block time, for example 6 seconds per block, there is a limited number of lines of code can be run during that time frame. When writing a pallet function, the developer is responsible for calculating its computational complexity, which is called weight. The process of determining that complexity, or simply put, time cost is called benchmarking.

Release Benchmarks

• Benchmarks are run again for each Release using Option 2: "Referenced Hardware"
• Release benchmarks are not merged back into main
• Benchmarks should still be run on your branches as needed so that PRs highlight changes in weights

Running Benchmarks

There are 2 options to run benchmarks:

• Locally on a developer laptop
• Remotely on referenced hardware in the cloud

1. Locally

❗ DO NOT commit weights yourself!

make benchmarks

It can be helpful to temporarily lower the number of iterations when testing benchmarks locally to decrease the time it takes to run them.

2. Referenced Hardware (GitHub Actions)

---

❗ ATTENTION:

1. DO NOT commit weights yourself! They will be auto-committed by CI job upon completion.
2. Currently, there is an issue with the benchmark runner not having benchmark label when it comes online. This label needs to be manually assigned by someone who has access to the Settings > Actions> Runners page before "Run Benchmarks" job can be picked up.

---

To trigger running benchmarks on referenced hardware in the cloud:

1. Open a PR and ensure that it is up to-date with main and ready for merging.
2. Request a core developer to run benchmarks.
3. A core developer will need to push a new PR commit with a commit message containing [run-benchmarks <pallets|all>] suffix in order to trigger benchmarks. Any text prior to [run-benchmarks <pallets|all>] is optional and can be omitted for empty commits. For example, this command:
   • git commit -m 'Run benchmarks [run-benchmarks msa,schemas]' – commits local changes and runs benchmarks for given pallets once commit is pushed to remote
   • git commit -m 'Run benchmarks [run-benchmarks all]' – commits local changes and runs benchmarks for given pallets once commit is pushed to remote
   • git commit -m 'Run benchmarks [run-benchmarks msa,schemas]' --allow-empty – once commit is pushed to remote, runs benchmarks for given pallets when there are no changes to commit
   • git commit -m '[run-benchmarks msa,schemas]' --allow-empty – same as above
   • git commit -m 'Run benchmarks [run-benchmarks all]' --allow-empty – once commit is pushed to remote, runs benchmarks for all pallets
4. Upon job completion, the updated weights will be committed back to the PR branch in a new commit and a comment will be added to the PR similar to this:

   Finished running benchmarks. Updated weights have been committed to the PR branch 
   in commit <commit_sha_link>.
   

2. Referenced Hardware (Jenkins, OBSOLETE)

❗ DO NOT commit weights yourself! They will be auto-committed by CI job upon completion.

To trigger running benchmarks on referenced hardware in the cloud:

1. Open a PR and ensure that it is up to date with main and ready for merging.
2. Request a core developer to run benchmarks.
3. Core developer will go to Jenkins (VPN Required) and kick off the build for that PR
4. Upon job completion, the updated weights will be committed back to the PR branch in a new commit.

Resources for Understanding and Writing Benchmarks

• https://docs.substrate.io/test/benchmark/
• https://www.shawntabrizi.com/substrate/substrate-weight-and-fees/
• https://github.com/shawntabrizi/substrate-graph-benchmarks
• https://github.com/paritytech/substrate/tree/master/utils/frame/benchmarking-cli/src/block