Lettuce Connection Disruptions

[bdchatham]

Hi there,

I am currently building an application that manages a connection pool to multiple Redis clusters. The application uses caching to store metadata related to the cluster management. Sometimes this metadata can become stale. It's possible for commands to be send to a cluster that no longer exists in this case.

The challenge I'm currently wrestling with is the ability to assert a node is no longer present, rather than there being a network partition, by only trying to make a network request to it. I know this is not possible, but I'm hoping someone can help me understand what the closest alternative is to this. For reference, I am using AWS ElastiCache to manage the underlying infrastructure of the Redis resources.

Thanks in advance!

[mp911de]

Ideally, you query your AWS deployment for the presence/absence of clusters and nodes if Redis-specific details aren't sufficient.

[bdchatham]

Hey Mark, thanks for the response.

Agreed on this point. We've considered this possibility, but for this very low-latency use-case, we're hesitant to add such a latent request to the code-path. It will effectively mean the failure of a customer call by making this check among other concerns, like thread utilization. I'm only thinking out-loud to help you understand my framing of the problem.

Ultimately, trying to understand if there is a particular strategy for handling such scenarios that you're aware of with expertise of Lettuce internals.

Perhaps using the connection failure events and only checking AWS ElastiCache state every ~5 retries would prevent widespread service degradation.