Cache read errors not recorded, and no retry for eager fetch

[stuhood]

As noticed previously (and then forgotten: oops): the remote cache does not record read errors when populating stdout/stderr on the response, or when eager fetching the content:

pants/src/rust/engine/process_execution/src/remote.rs

Lines 1393 to 1394 in 064036b

	// TODO: This should move inside the retry_call above, both in order to be retried, and
	// to ensure that we increment a miss if we fail to eagerly fetch.

The comment isn't quite right though: we can't trivially move that block inside of the retry_call without refactoring the relevant methods from String errors to something classifiable (which might be useful, but isn't necessarily the right scope to retry at, since local errors won't be retriable). Fixing the metric only involves moving all of the steps involved in the hit under an async block so that we don't call the hit "Ok" until eager_fetch has succeeded.

But instead, we should likely move to using tower's retry with tonic. This would have the advantage of applying to all gRPC methods in one fell swoop. That would remove local retry_call blocks and ease adding additional nesting here.

[tdyas]

First attempt at trying to use tower::retry::Retry: #12614

Doesn't compile, will need to discuss the types with someone.

[tdyas]

Apparently tonic does not work with tower::retry::Retry: hyperium/tonic#733