Reduce distributor memory usage when error volume is high

[damnever]

What this PR does:

Avoid nesting httpgrpcutil.WrapHTTPGrpcError.

This cluster has ~600 QPS, and ~400 of them are 2xx.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[harry671003]

If possible, could you add a heap profile after this fix for comparison?

[danielblando]

Nice optimization.

One question on your graph.
After the "upgrade with fix", I do see an increase in CPU, was that a rollback or scale down?

[damnever]

@harry671003 before the fix, our workload was OOM so it was easy to capture, today, our error rate is decreasing, the httpgrpcutil.WrapHTTPGrpcError is nearly invisible, so there is nothing interesting.

@danielblando yes, the distributor scaled down after the rollback, and the upgrade with the fix has the same replica count as before.

[yeya24]

Thanks!