[Feature request] Pod metrics

[NeQuissimus]

Similar to idea of the PodsHealth trait:
If there was a trait PodsMetrics or similar, which requires a way to define a load factor (maybe define it to be 0.0 <= x <= 1.0) for a Pod.

trait PodsMetrics {
  def loadFactor(podAddress: PodAddress): UIO[Float]
}

By using this load factor, a pod would report how "busy" it is (it could simply report the underlying machine's CPU usage or take into account other metrics).
Such a load factor could then be exposed further and, for example, be used to implement a Kubernetes HPA.
This would enable pods (in both the Kubernetes and Shardcake sense) to be added and removed automatically.

[NeQuissimus]

In addition, I could see logic that would implement this trait in such a way that communicates with pre-defined types of entities (maybe those extending an additional trait). Imagine my entity is a Kafka consumer and I would like to trigger starting up more pods when it falls behind (or reduce the number of consumers when everything is caught up).

[ghostdogpr]

I understand the need but I wonder if that needs to be in shardcake at all. Adding or removing pods is controlled by Kubernetes and it can already use metrics such as CPU. If application metrics need to be used, they can be exposed by an API. What would the shard manager do about it if it was exposed to it?

Btw we use HPA in production based on metrics like CPU usage, works exactly as you described.

[NeQuissimus]

I think I would mostly consider it a convenience feature, a way to standardize collecting and exposing load metrics.
I can see why it may not be necessary in Shardcake itself. A pod may want to be able to collect metrics from entities in order to expose some form of aggregation as a custom metrics (which can then become an input to the HPA).