Investigate Cache Usage and Memory

[manno]

With the switch to controller-runtime we can optimize the cache usage of the controllers. We also need to make sure there are no memory leaks in the controllers.

Exercise the controllers and investigate the cache usage with pprof (see https://github.com/rancher/fleet/blob/main/docs/performance.md).
Optimize the cache usage, e.g. by excluding sensitive or large resources with little re-use, like secrets, contents and bundles.

[manno]

The old continuous pprof code can be deleted. #2976
Pyroscope is easy to add to a dev build: https://grafana.com/docs/pyroscope/latest/configure-client/language-sdks/go_push/

• we can use partialmetadata in a few places, like this
• FieldIndexers might be useful, example
• helmcache does not need a clientset
• we can replace RetryOnConflict with a requeue for idempotent reconciles
• do metrics collectors grow over time?
• can we store helm manifest annotations on the secret instead of in the manifest, so we don't need to fetch/unpack all of them?

[manno]

No issues were found.