Fail fast when we can't start a cache for a client

[negz]

What problem are you facing?

xgql makes heavy use of caching. The first time a particular user asks for a particular kind of resource it needs to start and hydrate a new cache for that kind of resource, which it does by listing the resource and then starting a watch. The user's call blocks waiting for that cache to by hydrated, and times out after 5 seconds of waiting.

In some cases the cache layer can detect very quickly that the cache isn't going to start and therefore that the blocking request is going to timeout - e.g. because the caller doesn't have RBAC access to the requested type - but there's unfortunately no mechanism for the cache layer to communicate this to the caller.

How could xgql help solve your problem?

We could modify our cache layer to allow calls to fail fast in cases where it's obvious that the cache isn't going to hydrate - e.g. due to RBAC issues. We're currently using the controller-runtime caching machinery, mostly because it's well proven and was expedient to reuse.

I could see a few options here:

1. We implement our own cache that satisfies the controller-runtime client.Client interface, but that fails fast and surfaces failures up to the caller.
2. We raise an issue against controller-runtime to see whether they'd be interested in this behaviour. I don't recall whether the change would need to happen in controller-runtime or the lower level client-go machinery. This could be a bit of a hard sell because we're not really using the controller-runtime cache the way it was intended to be used.
3. We fall back rapidly when a cache takes too long to start and make an uncached call, which would fail in a more obvious manner if there were RBAC or network connectivity issues.