Update pallet weights every week

[ggwpez]

With changes to FRAME, the Rust compiler version, pallets and the benchmarker itself, we should try to update the pallet weights on a weekly basis to see regressions early and not just before a release.

Task: Update the weights of all pallets from all runtimes every week. It should re-use the already open MR and force-push there, to avoid spamming up new MRs if reviews are slow.

[bkchr]

Task: Update the weights of all pallets from all runtimes every week. It should re-use the already open MR and force-push there, to avoid spamming up new MRs if reviews are slow.

No no. These prs should be merged automatically if the weight changes are not a regression. Generally I would also expect if these kind of prs exist they should also get merged. No need to bring this stuff up and then it lays around.

[mordamax]

First iteration anyway is not going to go automatically, but through a review, at least to establish the baseline, as many of the weights either do not exist or very outdated

https://github.com/paritytech/polkadot-sdk/pull/6789/files
This workflow allows us to start (or restart) this as we want through workflow dispatch, and will create a PR with all runtimes recalculated diff + subweights report so it's easier to scan visually the outsiders.

The next step would be finding out how could we further automate it so it will require minimum of the developer intervention, ideally.

for now - My pov is that it should always go through at least 1 person, may be CI/RelEng or any core-dev. As i'd be very hesitant to trust any kind of tooling a weights pushed directly to master without human checking. Imagine CI/tooling bug which ends up with too low or crazy high numbers weights in master 🤯

May be adding some enhancements like bot pre-checking if it's all good or something is bad, or highlights the pallet outsiders, which would help developer to spot the degradation, and/or enable auto-merge in PR, but still a human to approve

[bkchr]

Imagine CI/tooling bug which ends up with too low or crazy high numbers weights in master 🤯

No one should use the weights in production...

[mordamax]

I guess I was wrong with adding subweights, as output for the whole monorepo is very large
Here's a PR it created #7280 but the CI is broken, as far as i see it's because of outdated hbs template?
@kianenigma @bkchr @ggwpez will you please check?

[ggwpez]

@iulianbarbu maybe? Just saw you do some testing with the omni bencher, so maybe you are interested.

[iulianbarbu]

Seems like #6789 attempts at fixing the hbs issue too. TBH not very familiar of what fix needs to be done, so leaving it with @mordamax .

[mordamax]

@iulianbarbu the hbs issue is related to #7204, however this also blocks the weekly benchmarks as they are not compiling either if we recalcualte all of them