Add Status Code Metrics for API Calls in the Agent

[Paramadon]

Description

Issue Overview

The agent currently lacks visibility into the status codes returned by its API calls, making it challenging to monitor API interactions and debug issues effectively.

Summary of Changes

This PR introduces functionality to collect and report status code metrics for the agent’s API clients, enhancing monitoring capabilities. The metrics cover the following APIs:

• PutRetentionPolicy
• DescribeInstances
• DescribeTags
• DescribeVolumes
• DescribeContainerInstances
• DescribeServices
• DescribeTaskDefinition
• ListServices
• ListTasks
• DescribeTasks
• CreateLogGroup
• CreateLogStream

---

Implementation Details

1. Handler Integration

   • For OpenTelemetry (OTel) API calls, the agent’s health is determined using the awsmiddleware component ID. The appropriate status code handler is attached to each client to enable metric collection.
   • For Telegraf metrics, handlers are integrated into plugins like Prometheus during client initialization, ensuring metrics are captured at runtime.

2. Metric Creation

   • The agent generates a JSON payload containing categorized status codes and sends it via PMD/PLE to the CloudwatchStatsHandler.
   • The CloudwatchStatsHandler processes the JSON and creates status code metric, enabling enhanced visibility.

---

Example Outputs

Example JSON Payload

Below is a sample JSON structure that categorizes status codes for various API calls:

{
  "codes": {
    "pmd": [1, 2, 3, 4, 5],
    "prp": [6, 7, 8, 9, 10],
    "di": [11, 12, 13, 14, 15],
    "dt": [16, 17, 18, 19, 20],
    "dv": [21, 22, 23, 24, 25],
    "dci": [26, 27, 28, 29, 30],
    "ds": [31, 32, 33, 34, 35],
    "dtd": [36, 37, 38, 39, 40],
    "dts": [41, 42, 43, 44, 45],
    "ls": [46, 47, 48, 49, 50],
    "lt": [51, 52, 53, 54, 55],
    "clg": [56, 57, 58, 59, 60],
    "cls": [61, 62, 63, 64, 65]
  }
}

Example Visualization

The screenshot below demonstrates the captured status codes in the monitoring interface:

---

Testing Details

• Configuration Used
  The agent was configured with the following agent.json file to validate metrics collection: (Also this is removed not but added some api calls so we can make sure we can get metrics for all the calls).

  {
    "agent": {
      "debug": true,
      "aws_sdk_log_level": "LogDebugWithHTTPBody",
      "region": "us-west-2"
    },
    "metrics": {
      "append_dimensions": {
        "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
        "ImageId": "${aws:ImageId}",
        "InstanceId": "${aws:InstanceId}",
        "InstanceType": "${aws:InstanceType}"
      },
      "metrics_collected": {
        "cpu": {
          "measurement": ["cpu_usage_idle", "cpu_usage_iowait", "cpu_usage_user", "cpu_usage_system"],
          "totalcpu": false,
          "metrics_collection_interval": 10
        },
        "disk": {
          "resources": ["*"],
          "measurement": ["used_percent", "inodes_free"],
          "metrics_collection_interval": 60
        },
        "diskio": {
          "resources": ["*"],
          "measurement": ["io_time", "write_bytes", "read_bytes", "writes", "reads"],
          "metrics_collection_interval": 60
        },
        "mem": {
          "measurement": ["mem_used_percent"],
          "metrics_collection_interval": 10
        },
        "netstat": {
          "measurement": ["tcp_established", "tcp_time_wait"],
          "metrics_collection_interval": 60
        },
        "swap": {
          "measurement": ["swap_used_percent"],
          "metrics_collection_interval": 10
        },
        "ethtool": {
          "interface_include": ["eth0", "eth1"],
          "metrics_include": ["bw_in_allowance_exceeded", "bw_out_allowance_exceeded", "pps_allowance_exceeded", "conntrack_allowance_exceeded", "linklocal_allowance_exceeded"]
        }
      }
    },
    "logs": {
      "metrics_collected": {
        "prometheus": {
          "prometheus_config_path": "/etc/cwagent/prometheus-config.yml",
          "log_group_name": "Prometheus",
          "emf_processor": {
            "metric_namespace": "CWAgent-Prometheus",
            "metric_unit": {
              "jvm_threads_current": "Count",
              "jvm_gc_collection_seconds_sum": "Milliseconds"
            }
          },
          "ecs_service_discovery": {
            "sd_frequency": "30s",
            "sd_target_cluster": "testCluster",
            "sd_cluster_region": "us-west-2",
            "sd_result_file": "/tmp/ecs-discovery.yml",
            "docker_label": {
              "sd_port_label": "ECS_PROMETHEUS_EXPORTER_PORT",
              "sd_metrics_path_label": "ECS_PROMETHEUS_METRICS_PATH",
              "sd_job_name_label": "job"
            }
          }
        }
      },
      "logs_collected": {
        "files": {
          "collect_list": [
            {
              "file_path": "/var/log/syslog",
              "log_group_name": "/aws/ec2/syslog",
              "log_stream_name": "{instance_id}",
              "retention_in_days": 30
            }
          ]
        }
      }
    }
  }

• Tests Performed

  • Verified metrics collection for each listed API.
  • Modified agent logic to ensure all status codes were reported.

---

Tested on EKS by changing agent contrib to this branch.

Ex: replace github.com/open-telemetry/opentelemetry-collector-contrib/processor/resourcedetectionprocessor => github.com/amazon-contributing/opentelemetry-collector-contrib/processor/resourcedetectionprocessor v0.0.0-20241212025412-4bd7a1e9deed

### Requirements

Before committing, complete the following:

1. Run make fmt and make fmt-sh.
2. Run make lint.

---

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution under the terms of your choice.

[Paramadon]

The indentation is because of our default indentation which was altered previously and this is correcting it. You can ignore the whitespace

[Paramadon]

adding host so we can grab the handler and attach them to the client.

[mitali-salvi]

Why is this PR being merged in CodeHandler branch and not main ?

[okankoAMZ]

I have 2 questions:

1. Have you tested this on EKS
2. Have you ran go race test on this

[okankoAMZ]

Testing looks good