Restoring from etcd database file

[tdudgeon]

The disaster recover docs (https://etcd.io/docs/v3.5/op-guide/recovery/) state that etcd can be restored from the database file (in the case that snapshot save cannot be run). But it does not provide instructions for doing this, only restoring from a snapshot that have been created using snapshot save.

Could someone advise on how to do this?
And update the docs to describe this?

[jmhbnz]

Can I please confirm you have the /member/snap/db file available to restore from? If so this can be used like a snapshot generated with something like etcdctl --endpoints $ENDPOINT snapshot save snapshot.db.

As mentioned in the recovery doc:

To restore a cluster, all that is needed is a single snapshot “db” file.

Snapshot integrity may be optionally verified at restore time. If the snapshot is taken with etcdctl snapshot save, it will have an integrity hash that is checked by etcdutl snapshot restore. If the snapshot is copied from the data directory, there is no integrity hash and it will only restore by using --skip-hash-check.

[alanbchristie]

We do have the /member/snap/db file. This is in a Kubernetes cluster but I am unable to run etcdctl in the etcd container (as the system is generally broken).

I will revisit this command and try and snapshot the db file somehow.

What's the significance of the $ENDPOINT value? If the original db was on 192.168.0.100 and the new (replacement) cluster/etcd service is on .101 what value do I use for the ENDPOINT when I run the save command?

[jmhbnz]

Hi @alanbchristie - If you have the /member/snap/db file from the failed container then you don't need to create a new snapshot, you can just use that file to restore from, either via the etcdutl snapshot restore command or via starting etcd with the --data-dir parameter.

You can follow a process roughly like below for the latter option:

1. Stop the etcd service on the old etcd node. Noting that they may already be stopped due to failure.
2. Copy the /member/snap/db file to new replacement etcd node.
3. Start the etcd service on that node with the snapshot file using the --data-dir option to specify the directory where the snapshot file is stored.
4. Verify that the etcd service has started successfully by checking its logs etc.

The --endpoint parameter value in etcdctl is used to specify the endpoint of an etcd server that etcdctl should communicate with, i.e. an endpoint specifies the IP address and port number where the etcd server can be reached.

[alanbchristie]

This seems to have worked - i.e. by simply stopping, replacing the db and restarting ... except I have simply moved to the next problem ... the database keystore (used by Kubernetes) has material (IP addresses) that relate to the original cluster. Not a urp[rise I guess. But it just means that etcd is happy but Kubernetes isn't. It's clear I cannot move the DB to a new cluster and expect applications to work (I guess I was foolish to even expect that I guess).

Anyway, thanks for your rapid response and help.

[alanbchristie]

Basically restoring an etcd DB on a node in an entirely new cluster just isn't possible. That, I suspect should have been obvious if I'd thought more about what was in etcd as well as trying to restore the etcd service.

[jmhbnz]

Hi @alanbchristie - It looks like you solved your issue in #15885 (comment) so I will mark this as resolved.

Just a side note for anyone that finds this discussion in future there is kubernetes specific etcd restoration documentation which relates to how to inform kubernetes of new etcd server ip addresses: https://kubernetes.io/docs/tasks/administer-cluster/configure-upgrade-etcd/#restoring-an-etcd-cluster

If the access URLs of the restored cluster is changed from the previous cluster, the Kubernetes API server must be reconfigured accordingly. In this case, restart Kubernetes API servers with the flag --etcd-servers=$NEW_ETCD_CLUSTER instead of the flag --etcd-servers=$OLD_ETCD_CLUSTER. Replace $NEW_ETCD_CLUSTER and $OLD_ETCD_CLUSTER with the respective IP addresses. If a load balancer is used in front of an etcd cluster, you might need to update the load balancer instead.

Just one final question, based on the restoration process you went through, is there something specific you felt was not covered in documentation? If so I would encourage you to raise an issue against the etcd-io/website repository and also invite you to open a pull request, we welcome new contributors 🙏🏻