[grafana-sampling] add sampling helm chart

[rlankfo]

This helm chart deploys a layered set of agents that can be used for OTLP load balancing, metrics generation, and trace sampling. This provides an opinionated solution that customers can use as somewhat of a "black box" on top of the agent. It does quite a bit of the river/flow configuration heavy-lifting.

[cyrille-leclerc]

This helm chart deploys a layered set of agents that can be used for OTLP load balancing, metrics generation, and trace sampling.

Doing RED & Svc Graph Metrics Generation + Sampling with the needed load-balancing for scalability is what I expected. I don't catch why there is also a K8s attributes processor, I think we don't want to enrich the incoming signals with any contextual metadata, we require metadata to be provided before entering the sampling box.

[LittleLadySunshine]

@cyrille-leclerc with @rlankfo, I might not be able to get this addressed until morning. FYI. @gouthamve @mar4uk

[mar4uk]

This is the architecture deployed by this helm chart:

The first layer (deployment) receives all OTLP signals and enriches them with k8s.* resource attributes. Then it exports metrics (app metrics) and logs to Grafana Cloud, and load-balances traces to the second layer with load-balancing exporter.

The second layer (statefulset) receives traces from the first layer does metrics generation and sampling, and exports generated metrics and sampled traces to Grafana Cloud.

So basically, the second layer is the "sampling box", it indeed receives enriched data

[cyrille-leclerc]

As discussed with @mar4uk , we don't see reasons to enrich incoming OTel signals with K8s Metadata.

The k8s.* enrichment, as well as the cloud.* and host.* enrichments, should be done before the Grafana Sampling layer as shown in the architecture diagram below

[mar4uk]

Shall we put this diagram (the one Cyrille added) to the DesignDoc?
So it would be clear that the sampling helm chart covers only the Sampling Box part of the diagram.

The data enrichment processors

• otelcol.processor.transform "add_resource_attributes_as_metric_attributes"
• otelcol.processor.transform "add_k8s_attributes"
• otelcol.processor.k8sattributes "default"
• resourcedetectionprocessor (later, when it is released. As a separate PR)

are going to be moved out of the Sampling Box and be a part of k8s-monitoring helm chart (they are already a part of the k8s-monitoring helm chart)

wdyt, Robbie?

[cyrille-leclerc]

The data enrichment processors ... are going to be moved out of the Sampling Box and be a part of k8s-monitoring helm chart (they are already a part of the k8s-monitoring helm chart)

Do we want to enrich meta-monitoring metrics and logs? I guess Grafana Agent's OTel Collector components publish meta-monitoring metrics similarly to what the Otel Collector does.

Note that meta-monitoring could be tackled in a subsequent milestone

[rlankfo]

Shall we put this diagram (the one Cyrille added) to the DesignDoc? So it would be clear that the sampling helm chart covers only the Sampling Box part of the diagram.

The data enrichment processors

• otelcol.processor.transform "add_resource_attributes_as_metric_attributes"
• otelcol.processor.transform "add_k8s_attributes"
• otelcol.processor.k8sattributes "default"
• resourcedetectionprocessor (later, when it is released. As a separate PR)

are going to be moved out of the Sampling Box and be a part of k8s-monitoring helm chart (they are already a part of the k8s-monitoring helm chart)

wdyt, Robbie?

I'll remove these for now but my gut feeling is that not everyone who uses this helm chart will use the k8s monitoring helm chart as well and eventually we may get requests to make some of these processors configurable in the chart.

[rlankfo]

Do we want to enrich meta-monitoring metrics and logs? I guess Grafana Agent's OTel Collector components publish meta-monitoring metrics similarly to what the Otel Collector does.

I don't think we want to enrich any debug metrics coming out of the agent. I believe these metrics would need to be scraped from the agents /metrics endpoint.

[LittleLadySunshine]

Let's make sure we keep meta monitoring on our horizon for future work. cc: @rlankfo

[petewall]

I'd love to see a quick README to briefly explain what's going on here.

Otherwise, looking good.

[mar4uk]

I'll remove these for now but my gut feeling is that not everyone who uses this helm chart will use the k8s monitoring helm chart as well and eventually we may get requests to make some of these processors configurable in the chart.

We are going to advise users to use k8s monitoring helm chart if they run their applications in Kubernetes. If they prefer to not use k8s monitoring helm chart anyway (I'm wondering what could be the reasons for not using it), they would still deploy their custom Grafana Agent or Collector. We probably can provide some recommended configurations for this custom Agent/Collector.

The downside of having enhancement processors in the sampling box in case when users don't use k8s monitoring helm chart is we will enhance only traces. Users still would need to enhance metrics and logs somewhere before the sampling box

[rlankfo]

We are going to advise users to use k8s monitoring helm chart if they run their applications in Kubernetes. If they prefer to not use k8s monitoring helm chart anyway (I'm wondering what could be the reasons for not using it), they would still deploy their custom Grafana Agent or Collector. We probably can provide some recommended configurations for this custom Agent/Collector.

We provide the recommended configurations but we don't provide a way to deploy them. The suggestion was to make the processors configurable in this helm chart when users do not deploy k8s monitoring. There could be several reasons such as cost, having other monitoring solutions implemented for k8s, etc.

The downside of having enhancement processors in the sampling box in case when users don't use k8s monitoring helm chart is we will enhance only traces. Users still would need to enhance metrics and logs somewhere before the sampling box

Not necessarily. The way it was originally configured, it allowed to send metrics and logs through the first layer as well. For now, we've chosen to remove this functionality in any case but if there are more comments around it lets iterate on the doc.

[hedss]

Hey folks. I've been pointed at this chart draft, and want to raise a few comments, not about the code itself but more a general conversation around usage.

We now have several different Agent charts for different purposes, including the 'vanilla' agent install and the k8s one. An issue with layering more charts in is that we're now starting to advise users to potentially put in entire pipelines of Agents in their infrastructure to get appropriate signals into our backend. This becomes problematic, and it's something we've seen in GTM.

Could some of this work not be carried out using Agent Modules?

The alternative is a user having to pick the chart that does the 'heaviest lifting' for them to start with, and start to override config, or at worst rewrite bits of charts, to actually do what they need. I would very much welcome a conversation here, or elsewhere, to discuss this.

[cyrille-leclerc]

Thanks for reaching out @hedss, with great pleasure to have a conversation, I'll DM you.

[mar4uk]

Looks good! Left a couple of comments, I think they should be fixed before merging.

Did you exclude logs and metrics pipelines because they should be handled by a separate agent?

Let me know if you need help with the diagram update (logs and metrics arrows should be removed, right?)

[rlankfo]

Looks good! Left a couple of comments, I think they should be fixed before merging.

Did you exclude logs and metrics pipelines because they should be handled by a separate agent?

Let me know if you need help with the diagram update (logs and metrics arrows should be removed, right?)

Thanks Irina! I'll make these updates. I would always appreciate any help with the diagram. It should only support traces.

[mar4uk]

LGTM!