Kubernetes lease name truncation can lead to non-unique lease names

[an-tex]

Versions used

Versions:
Akka: 2.6.13
Akka Management: 1.0.10

Expected Behavior

When using a kubernetes lease for cluster sharding (https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html#lease), the kubernetes lease names should be unique.

Actual Behavior

In my case I've enabled a projection using a sharded daemon process, using a kubernetes lease for coordination. The lease names end up being prefixed with application-shard-sharded-daemon-process-[...] (42 characters) and being truncated to 63 characters, this leaves only 22 characters for the actual projection name + tag. When truncated, the tag is the first to disappear, this generates the same lease name for different sharding processes. This in turn leads to an error loop with the following logs:

Lease application-shard-sharded-daemon-process-[...] requested by client [email protected]:25520 is already owned by client. Previous lease was not released due to ungraceful shutdown.

Conflict during heartbeat to lease LeaseResource(Some([email protected]:25520),8609152,1622013422731). Lease assumed to be released

sharded-daemon-process-[...]: Shard id [2] lease lost, stopping shard and killing [1] entities.

[an-tex]

About possible solutions:

1. When truncating, add a shortish hash suffix based on the full name
2. Use multiple domain segments, as the 63 character limit only counts for each segment. The whole name can be up to 253 characters. Meaning a lease name like application.shard-sharded-daemon-process.projectionName.tag would mitigate the limits.

[manuelp]

This is a serious problem, as:

• This is not documented and implicit. We found out about this problem the hard way.
• Most of the characters are used by Akka's internals.
• Truncation is transparent, there is no way to know beforehand if a truncation will happen.
• More importantly, when a lease name is truncated there can be implicit naming collisions that cause the system to essentially stop working.

I understand that this is a limitation imposed by k8s, and Akka has no way around that. But the current solution is IMHO fragile and prone to error, so not a real solution.

@an-tex proposed some workarounds. A fixed-length encoding could be a real solution, at the expense of lease names legibility. But I think that an imperfect working system is better than a "legible" (not really the case with truncated lease names) but broken one.

Are we missing something? There is workaround that we could employ right now? How this is working in production for others?

[octonato]

The prefix is a concatenation of different bits of information. As described here:

https://doc.akka.io/docs/akka/current/typed/cluster-sharding.html#lease

To use a lease for sharding set akka.cluster.sharding.use-lease to the configuration location of the lease to use. Each shard will try and acquire a lease with with the name -shard-- and the owner is set to the Cluster(system).selfAddress.hostPort.

Another alternative solution could be to make it configurable. But I think only the <actor system name>-shard can be made configurable. The <type name>-<shard-id> portion will need to be kept around.

What about going for solution 2 as suggested by @an-tex, with failure in case the final name is larger than 253 chars?

We can also apply solution 1 in case it surpass 253 chars, but the problem I see with hashing is that it will make it harder to follow logs.

[an-tex]

@manuelp my "workaround" is NOT to use the kubernetes lease implementation for sharding & singleton :/ it's still fine to use with SBR.

I'd also opt for my 2. solution for the same reasons as @octonato mentioned, with either failure or error log & fallback to default (no lease).

[manuelp]

What about going for solution 2 as suggested by @an-tex, with failure in case the final name is larger than 253 chars?

Early failure would be preferable to silent truncation. At least we'd have an exception that highlights the underlying problem instead of a failing cluster that can't acquire leases.

We can also apply solution 1 in case it surpass 253 chars, but the problem I see with hashing is that it will make it harder to follow logs.

Certainly it'd be prefereable to have readable names, but in that case a working system with unreadable lease names it's still preferable to a failing system with readable ones IMHO.

my "workaround" is NOT to use the kubernetes lease implementation for sharding & singleton :/ it's still fine to use with SBR.

Yes, previously we weren't using the leasing part, but it's useful (in our context at least).

I'd also opt for my 2. solution for the same reasons as @octonato mentioned, with either failure or error log & fallback to default (no lease).

Readable names and early failures with fallback to no leasing seems to me the best solution so far, if possible.

[patriknw]

Fail early sounds good to me. Better to fail hard than error and fallback, because some don't even pay attention to errors.

Changing to multiple domain segments sounds good, but it would have to be an opt-in configuration because otherwise we would break rolling updates of existing users.

[patriknw]

Fail early check added in #1194