Allow ReplicaSet and Job metadata generators to use partial meta objects

[swiatekm]

We only use metadata from Jobs and ReplicaSets, but require that full resources are supplied. This change relaxes this requirement, allowing PartialObjectMetadata resources to be used. This allows callers to use metadata informers and avoid having to receive and deserialize non-metadata updates from the API Server.

See elastic/elastic-agent#5580 for an example of how this could be used. I'm planning to add the metadata informer from that PR to this library as well. Together, these will allow us to greatly reduce memory used for processing and storing ReplicaSets and Jobs in beats and elastic-agent.

This is will help elastic/elastic-agent#5580 and elastic/elastic-agent#4729 specifically, and elastic/elastic-agent#3801 in general.

[pkoutsovasilis]

LGTM! eager to see the impact of using PartialObjectMetadata on the mem footprint

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 30ded63

History

• 💚 Build #254 succeeded b910b89
• 💔 Build #253 failed 088bf6d
• 💔 Build #252 failed ecd4ff2
• 💔 Build #251 failed 923e4e0
• 💔 Build #249 failed c0dd67c

[ycombinator]

LGTM! eager to see the impact of using PartialObjectMetadata on the mem footprint

@swiatekm Is there some way for you to quantify this impact (before vs. after), perhaps as a function of the number of ReplicaSets and Jobs?

[swiatekm]

LGTM! eager to see the impact of using PartialObjectMetadata on the mem footprint

@swiatekm Is there some way for you to quantify this impact (before vs. after), perhaps as a function of the number of ReplicaSets and Jobs?

Hard to do so in general, because the size of the resource in memory can heavily depend on user-provided metadata. For example, it's possible to set multiple annotations of up to 256kb each on it. In a test cluster I saw agent use 80% less memory on ReplicaSets specifically, but in real production clusters I'd expect the gains to be more substantial.

If we're talking about the total agent memory consumption, then that obviously depends on what inputs are enabled, but with the standalone Kubernetes manifest we provide, in a test cluster of 3 Nodes, with ~7500 ReplicaSets, I saw a reduction of ~120 Mb. You can see more details here: elastic/elastic-agent#5623.