Add a blog post to summarize the issue in 3.5 to 3.6 upgrade

[ahrtr]

The blog is to summarize the upgrade issue etcd-io/etcd#19557. We need to broadcast the blog post to increase awareness, so that users take action before upgrading to v3.6.0[-rc.x].

cc @fuweid @ivanvc @jberkus @jmhbnz @serathius @siyuanfoundation @spzala @wenjiaswe

[ahrtr]

also cc @neolit123

[serathius]

Should user care at all if v3.6 will come with a complete fix and no user action is needed? or maybe I didn't understood etcd-io/etcd#19636

[ahrtr]

Should user care at all if v3.6 will come with a complete fix and no user action is needed? or maybe I didn't understood etcd-io/etcd#19636

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

[ahrtr]

Should user care at all if v3.6 will come with a complete fix and no user action is needed? or maybe I didn't understood etcd-io/etcd#19636

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

Also it's part of the fixes. The summary should be complete.

All the links are just for experienced users references. I don't think normal users will understand the details of the PRs without expertise of etcd. It's exactly the reason why we need to ensure the blog is clear and easy to understand.

Also we have a "TL; DR" section for entry-level users or anyone who don't care about the details.

[serathius]

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

With etcd-io/etcd#19636 would it just work?

[ahrtr]

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

With etcd-io/etcd#19636 would it just work?

Obviously it won't work. Please see the the first case as mentioned in
If the etcd cluster has already been affected by the issue, there are two possible outcomes

[ahrtr]

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

With etcd-io/etcd#19636 would it just work?

Obviously it won't work. Please see the the first case as mentioned in If the etcd cluster has already been affected by the issue, there are two possible outcomes

To be clearer, the etcd-io/etcd#19636 is only to address the second case as mentioned in the two possible cases if users still upgrade from 3.5.1-3.5.19 to 3.6.0.

[jberkus]

One tiny grammatical change, but also I think we really should change the title to be more grabbing.

[spzala]

Thanks @ahrtr !! I have added few small comments inline.

[ahrtr]

Thanks both @jberkus and @spzala for the review!

Basically addressed all comment, PTAL, thx

[spzala]

Thanks for quickly addressing the review comments @ahrtr !!

[siyuanfoundation]

With etcd-io/etcd#19636 would it just work?

etcd-io/etcd#19636 would just work if v2store snapshot is saved. If not, we have to assume the v3store is correct, and prevent too many learners.

[serathius]

It answers the question "What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?".

With etcd-io/etcd#19636 would it just work?

Obviously it won't work. Please see the the first case as mentioned in If the etcd cluster has already been affected by the issue, there are two possible outcomes

Oh, I forgot that in v3.6 the etcd will bootstrap membership from v3 storage. Sorry, I'm 100% focused on KubeCon. This means that etcd might have incorrect information until fully we process local WAL.

With etcd-io/etcd#19636 in wouldn't a complete fix just require on more step? When patching storage during bootstrap also read WAL for any PromoteLearner proto, and we are done? No need to communicate, no need for user action, etc.

[ahrtr]

When patching storage during bootstrap also read WAL for any PromoteLearner proto, and we are done?

Yes. It's exactly the reason why we sync the v3store not only on bootstrap (inside (*bootstrappedCluster) Finalize(...)", but also during the apply (in case the learner promotion is still in the WAL records).

No need to communicate, no need for user action, etc.

Right, no further action is needed from users. We just need to clarify it in the blog to avoid any potential confusion or questions.

Overall, users are only required to upgrade to 3.5.20 (or a higher version) before upgrading to 3.6.0. We also clarify what will happen if users do not follow this guide (we guarantee that we won't silently swallow any errors).

[siyuanfoundation]

nit:

Suggested change

	it was discovered in Kubernetes' workflow test. To address this gap, we added a similar e2e
	test via [19634][], which was also backported to release-3.6 via [19662][].
	it was discovered in Kubernetes' workflow test. To address this gap, we added a Kubernetes style upgrade e2e
	test via [19634][], which was also backported to release-3.6 via [19662][].

[ahrtr]

Let's try not to create new concept "Kubernetes style upgrade" to avoid any unnecessary confusion/questions, also avoid wordy explanation.

I used "Kubeadm style upgrade" in the description of etcd-io/etcd#19557. It's OK in that informal case, but suggest not for this formal (and to be widely broadcasted) blog.

[ivanvc]

LGTM. I left a comment with a minor nit. Thanks, @ahrtr.

[ahrtr]

@jberkus we will release etcd v3.6.0-rc.3 tonight. I am going to merge this PR right after the release being out.

Do you have any further comments?

[jberkus]

Nope, it's good to go

/hold

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, ivanvc, jberkus, neolit123, siyuanfoundation, spzala

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahrtr,ivanvc,jberkus,spzala]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ivanvc]

/unhold

v3.6.0-rc.3 is out :)

[ahrtr]

Thanks. @ivanvc

[ahrtr]

Thanks all for the review.

Do you think we should broadcast this blog now (given v3.6.0-rc.3 is already out) or wait until v3.6.0 is out? @fuweid @jberkus @jmhbnz @ivanvc @serathius @spzala @wenjiaswe

https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue/

[spzala]

Thanks all for the review.

Do you think we should broadcast this blog now (given v3.6.0-rc.3 is already out) or wait until v3.6.0 is out? @fuweid @jberkus @jmhbnz @ivanvc @serathius @spzala @wenjiaswe

https://etcd.io/blog/2025/upgrade_from_3.5_to_3.6_issue/

Considering the blog post is public and upgrade to rc versions can potentially hit the issue covered in the blog, I think it's probably a good idea to broadcast (after KubeCon). It can also serve as a good reminder/heads-up for v3.6.0, and at that time we can amplify more. Just thoughts if others agree. Thanks @ahrtr !!

[ahrtr]

Thanks @spzala for the feedback, which makes sense.

I will call this out in KubeCon maintainer session regardless. Leave it to @jberkus to broadcast?

[spzala]

Thanks @spzala for the feedback, which makes sense.

I will call this out in KubeCon maintainer session regardless. Leave it to @jberkus to broadcast?

Thanks @ahrtr !!