Benchmarking

In order to achieve a certain block time, for example 6 seconds per block, there is a limited number of lines of code can be run during that time frame. When writing a pallet function, the developer is responsible for calculating its computational complexity, which is called weight. The process of determining that complexity, or simply put, time cost is called benchmarking.

Release Benchmarks

• Benchmarks are run again for each Release On Referenced Hardware
• Release benchmarks are not merged back into main
• Benchmarks should still be run on your branches as needed so that PRs highlight changes in weights

Running Benchmarks

There are 2 options to run benchmarks:

• Locally on a developer laptop
• Remotely on referenced hardware in the cloud

1. Locally

❗ DO NOT commit weights yourself!

make benchmarks

It can be helpful to temporarily lower the number of iterations when testing benchmarks locally to decrease the time it takes to run them.

2. On Referenced Hardware

❗ DO NOT commit weights yourself! They will be auto-committed by CI job upon completion.

To trigger running benchmarks on reference hardware in the cloud:

1. Open a PR and ensure that it is up to date with main and ready for merging.
2. Request a core developer to run benchmarks.
3. Core developer will go to Jenkins (VPN Required) and kick off the build for that PR
4. Upon job completion, the updated weights will be committed back to the PR branch in a new commit.

Resources for Understanding and Writing Benchmarks

• https://docs.substrate.io/test/benchmark/
• https://www.shawntabrizi.com/substrate/substrate-weight-and-fees/
• https://github.com/shawntabrizi/substrate-graph-benchmarks
• https://github.com/paritytech/substrate/tree/master/utils/frame/benchmarking-cli/src/block