Drop original metrics support added for all metrics.

[musa-asad]

Description of the issue

Public docs state that the drop_original_metrics feature is supported, but, when we try to run it, the config translation states it isn't allowed: Under path : /metrics/metrics_collected/statsd | Error : Additional property drop_original_metrics is not allowed.

Description of changes

I allow drop_original_metrics for the statsd metric as well as other metrics that didn't have it (collectd and ethtool) and adjusted the CW exporter translator code to allow for this.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

I first replicated the issue and then tried running CWAgent with the following config:

{
    "metrics": {
        "namespace": "DropTest-1",
        "aggregation_dimensions": [
            ["Group"]
        ],
        "metrics_collected": {
            "statsd": {
                "service_address": ":8125",
                "metrics_collection_interval": 1,
                "metrics_aggregation_interval": 5,
                "drop_original_metrics": ["should_drop"]
            }
        }
    }
}

The relevant errors are as follows (when we try using drop_original_metrics for all metrics collected):

Under path : /metrics/metrics_collected/ethtool | Error : Additional property drop_original_metrics is not allowed
Under path : /metrics/metrics_collected/collectd | Error : Additional property drop_original_metrics is not allowed
Under path : /metrics/metrics_collected/statsd | Error : Additional property drop_original_metrics is not allowed

After that, I re-ran the configuration and the error didn't persist.

Then, I ran echo "should_drop:123|c|#Group:Dropped" | nc -u -w1 127.0.0.1 8125 and echo "should_not_drop:123|c|#Group:Not_Dropped" | nc -u -w1 127.0.0.1 8125.

As expected, the should_drop didn't show up in the CW console in its original form, but should_not_drop did. However, they both showed up in their aggregated forms.

We also see this in the yaml file:

exporters:
    awscloudwatch:
        drop_original_metrics:
            should_drop: true

collectd
The drop_cpu_value didn't show up in the CW console in its original form, but show_cpu_value did. However, they both showed up in their aggregated forms.

ethtool
The ethtool_queue_0_rx_bytes didn't show up in the CW console in its original form, but ethtool_queue_0_tx_bytes did. However, they both showed up in their aggregated forms.

Additionally, I updated the unit tests to accommodate the changes made in this PR.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[github-actions]

This PR was marked stale due to lack of activity.

[sky333999]

Mind adding a screenshot of what the metrics look like in CW console?
I would have expected an append_dimensions for the InstanceId so Im curious how it worked without that.

[mitali-salvi]

Could you verify that that even though the original metric is being dropped, its still captured as part of the aggregation metric ?

[mitali-salvi]

Will this parameter still drop metrics even if the aggregation_dimensions parameter is not being used ?

[musa-asad]

Will this parameter still drop metrics even if the aggregation_dimensions parameter is not being used ?

Yeah, it will still drop those metrics.