Add experimental reconcile/rest API metric sanity test to E2E Tests

[jgwest]

What does this PR do / why we need it:

• This PR adds logic to sanity test the behaviour of the operator during the tests:
  • We check the (prometheus) metrics coming from the 'localhost:8080/metrics' endpoint of the operator.
  • For example, if the reported # of Reconcile calls was unusually high, this might mean that the operator was stuck in a Reconcile loop
  • Or, if the number of REST client POST requests (e.g. k8s objection creation) was equal to the number of PUT request (e.g. k8s object update), this may imply we are updating .status or .spec of an object too frequently.
• After this was enabled, I noticed we were reconciling too often, and I discovered that in many cases we were incorrectly detecting cases where nothing had changed, as something having changed.
  • This was due to known issues with comparing maps using reflect.DeepEqual, so I switched over to use a substitute function for this case.
  • I also noticed we were stripping out rbac.authorization.k8s.io/* as a user-defined label, when it is in fact not user-defined, so added it to ignore list.

Have you updated the necessary documentation?

• Documentation update is required by this PR, and has been updated.

Which issue(s) this PR fixes:
N/A

[chetan-rns]

How are these thresholds calculated? Do we compare the metrics before and after and use them as the baseline?

[jgwest]

These are just arbitrary thresholds (guesses) about what a 'normal' set of metrics should be:

• If we find that they are too restrictive, we should increase them.
• If we find that they are too lax, we should decrease them.

But these numbers aren't scientific by any means, they are just a guess that can serve as a starting point to allow us to refine them.

[chetan-rns]

LGTM. Thanks @jgwest