etcd client retry behavior

[marriner-stripe]

I have a few questions around retry behavior. We are planning on using etcd in scenarios where there are many clients interacting with a single cluster.

We are worried about the default retry behavior on Gets(100 retries on Service Unavailable errors) causing self-sustaining overloads and are considering disabling/changing retries.

1. Is there any way to change the retry policy for the client other than disabling retries entirely? Even hacky approaches would work for us for now! The best approach we were able to find was to change the client to the non-retrying version, e.g.

etcdClient.KV = etcdv3.NewKVFromKVClient(etcdpb.NewKVClient(etcdClient.ActiveConnection()), etcdClient)

2. Are there any known downsides to disabling retries, other than lower availability on a flaky network? Were the values of this specific retry policy chosen intentionally or are they generic?

[lavacat]

I don't think there is a way to change policy. Looking at git blame, seams like @ptabor updated TODO to use RetryPolicy

	- // TODO: Replace all of clientv3/retry.go with interceptor based retry, or with
	- // https://github.com/grpc/proposal/blob/master/A6-client-retries.md#retry-policy
	- // once it is available.
	+ // TODO: Replace all of clientv3/retry.go with RetryPolicy:
	+ // https://github.com/grpc/grpc-proto/blob/cdd9ed5c3d3f87aef62f373b93361cf7bddc620d/grpc/service_config/service_config.proto#L130

That's probably best direction to follow if you want to contribute this change.

Another option is to make defaultUnaryMaxRetries, defaultStreamMaxRetries, defaultBackoffWaitBetween, defaultBackoffJitterFraction configurable. Similar issue #13344

[lavacat]

We are worried about the default retry behavior on Gets(100 retries on Service Unavailable errors) causing self-sustaining overloads and are considering disabling/changing retries.

Have you done load testing for your use case? What types of Unavailable errors are you seeing? And what types of repeatable requests generate them?