Remember there is no perfect failure detector

[SchSeba]

@MikeSpreitzer I move the issue here to have a better visibility

Leader election does not guarantee that only one instance thinks it is leader at a time. While normally at most one is active at a time, there are corner cases in which there can be multiple active at once. This is not a bug in leader election that we can expect to be fixed; it is a consequence of a fundamental problem in distributed systems.

The kubernetes API machinery supports a very limited repertoire of ACID transactions. One is to create an object if and only if there is not already an object of the same kind, namespace, and name. That means we can use an object as a lock on any sort of thing that can be named.

In https://github.com/MikeSpreitzer/kube-examples/tree/add-kos/staging/kos#the-ipam-controller you will see an example of using this to allocate IP addresses.