What is the readiness criteria for etcd releaes ?

[pchan]

I would like to know the readiness criteria for an etcd release. The github checks should pass is given, but are there any specifics on what other kubernetes based testing that is done.

[serathius]

Usually we run K8s Scalability tests.

[pchan]

Can you please be more specific ? Do you run the k8s-master-blocking tests or all the sig-scalability (tests). Only the 100 node tests are considered as master-blocking in k8s, the 5000 nodes ones are only considered master-informing and don't seem to be run for k8s releases.

[serathius]

5000 node k8s tests are too costly to be run on every PR. They are run periodically non blocking k8s development, however any regressions will block k8s release. Historically I worked with Scalability team to run 5000 node tests for etcd minor releases and patch releases that expect performance impact.

[pchan]

I also noticed that the release document referred to in the release script is obsolete, an example being the performance testing section [1]. Are they replaced by benchmarks ? I can work on updating upstream documentation to make it easy for release maintainers.

[1]

Step 4: Performance testing (for minor release)
Each minor release gets new features. Once a new release branch is created, stop the feature development and:

Make sure etcd is resilient from failure injections/failpoints
etcd functional tester / failpoints
http://dash.etcd.io/dashboard/db/functional-tests?orgId=1
http://dash.etcd.io/dashboard/db/functional-tests-failpoints?orgId=1
This has been manually redeployed with etcd master branch, whenever team has some spare time. And it has been pretty stable. When a release candidate is being tested, there should never be a failure (e.g. do not release until etcd tolerates >10k rounds).
kubemark tests against actual 3-node etcd cluster with quorum-read on
http://dash.etcd.io/dashboard/db/test-etcd-kubernetes?orgId=1
In-depth performance analysis with dbtester
Comparing with Zookeeper and Consul is not needed anymore
Compare with previous etcd release to find any regression

[serathius]

I don't thi did any benchmarking since v3.0 documentation pages are just copied from release to release without any changes since then (https://etcd.io/docs/v3.1/benchmarks/etcd-3-demo-benchmarks/, https://etcd.io/docs/v3.2/benchmarks/etcd-3-demo-benchmarks/, https://etcd.io/docs/v3.3/benchmarks/etcd-3-demo-benchmarks/, https://etcd.io/docs/v3.4/benchmarks/etcd-3-demo-benchmarks/, https://etcd.io/docs/v3.5/benchmarks/etcd-3-demo-benchmarks/).

I was not even aware of this page in etcd.io. I think the reason was that there were not documented. This is the latest release documentation https://github.com/etcd-io/etcd/blob/main/Documentation/contributor-guide/release.md. Still some sections are already outdated.

I think it would be really useful to restore process of release qualification and benchmarking as part of release process. I was planning to add release qualification after #15102.

If you want to work on this, feel free to file an feature request issue so we can discuss the details.
https://github.com/etcd-io/etcd/issues/new?assignees=&labels=type%2Ffeature&template=feature-request.yml