Aggregating timer across tags

[maneeshbhunwal123]

I started looking into tags supported in 5.x version of metric.
But after doing some experiments, I have come to realise that it is not what i expected it to be.

In my understanding, we can add tags while submitting the metrics, and we can use these tags to generate appropriate name for metric. But there is no aggregation performed, across tags for example

lets say we publish 2 metrics with tags
metric1, tag={"userType": "Premium"}
metric1, tag={"userType": "Normal"}
IMO we should have been publishing metrics where both these metrics were aggregated, and optionally a way to segregate them.
metrics1 = aggregate of (metric1, tag={"userType": "Premium"}, metric1, tag={"userType": "Normal"})
metrics1NormalUser = metric1, tag={"userType": "Normal"}
metrics1PremiumUser = metric1, tag={"userType": "Premium"}

but look at implementation i don't see it being done. Can someone explain me what tags are trying to solve for and what is out of scope?

[joschi]

@maneeshbhunwal123 I'm not actively working on the release/5.0.x branch of Dropwizard Metrics. Please see #1704 for the current progress.

This being said, I think the Dropwizard Metrics library shouldn't perform any additional aggregation of tagged metrics up-front but leave it to the downstream receiver to do this.

@dropwizard/metrics Maybe someone else wants to chime in?

[baharclerode]

Dropwizard Metrics library shouldn't perform any additional aggregation

+1 to this!

I can offer from experience of helping hundreds of teams manage 1000's of dropwizard applications in a large company that the fundamental issue with something like this is scalability. The volume of metrics that even a medium-sized dropwizard application produces in a production context can very easily overwhelm metric ingestion pipelines.

what tags are trying to solve for

As your example demonstrated @maneeshbhunwal123, tagging itself doesn't really change the metrics that are reported. It's just a fancy API for standardizing names. Even with metrics v4 today, we still tag our metrics and emit the same volume of metrics, it's just by convention instead of config. A tagging API solves the problem of libraries and code written by multiple entities denoting tags in the same way. Without it, everyone still produces tagged metrics, and embeds those tags in the metric name, but in slightly different ways and you have to post-process metric names into a standard format after collection to extract tags.

But there are an exponential explosion of tag combinations when you start trying to aggregate, which each requires memory, time, and CPU-- you can't have every possible aggregation, so you have to force the developer to pick and choose which aggregations are important.

Given that:

1. You want to minimize the impact of any monitoring systems on an application
2. You want to minimize network bandwidth required to collect monitoring data
3. If we can automatically calculate aggregates from component metrics locally, we can do the same in the metrics pipeline
4. We require

I would very much argue it's not the place of dropwizard to perform these aggregations. Metrics processing pipelines today already have fantastic support for aggregating metrics in all sorts of ways, and we shouldn't try to duplicate that here.