Add integ test for Nvidia GPU in EKS

[movence]

Description of changes

Add integ test for GPU with EKS. Highlights of this PR:

• There will be no GPU node used for the test, since the test will be using httpd to mock dcgm exporter
• httpd server will serve /metrics endpoint with a static prometheus data file. All metric values are 1
• Terraform creates CA and TLS certs then mounts them to the agent and httpd pods using k8s secrets
• The validation logic will check expected metrics with their corresponding dimension sets

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Integ test run: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/8542742082

null_resource.validator (local-exec): 2024/04/03 10:53:45 >>>>>>>>>>>>>><<<<<<<<<<<<<<
null_resource.validator (local-exec): 2024/04/03 10:53:45 >>>>>>>>>>>>>>Successful<<<<<<<<<<<<<<
null_resource.validator (local-exec): 2024/04/03 10:53:45 ==============EKS_GPU_NVIDIA==============
null_resource.validator (local-exec): 2024/04/03 10:53:45 ==============Successful==============
null_resource.validator (local-exec): ClusterName-ContainerName-FullPodName-GpuDevice-Namespace-PodName   Successful
null_resource.validator (local-exec): container_gpu_memory_total                                          Successful
null_resource.validator (local-exec): container_gpu_memory_used                                           Successful
null_resource.validator (local-exec): container_gpu_power_draw                                            Successful
null_resource.validator (local-exec): container_gpu_temperature                                           Successful
null_resource.validator (local-exec): container_gpu_utilization                                           Successful
null_resource.validator (local-exec): container_gpu_memory_utilization                                    Successful
null_resource.validator (local-exec): ClusterName-FullPodName-GpuDevice-Namespace-PodName                 Successful
null_resource.validator (local-exec): pod_gpu_memory_total                                                Successful
null_resource.validator (local-exec): pod_gpu_memory_used                                                 Successful
null_resource.validator (local-exec): pod_gpu_power_draw                                                  Successful
null_resource.validator (local-exec): pod_gpu_temperature                                                 Successful
null_resource.validator (local-exec): pod_gpu_utilization                                                 Successful
null_resource.validator (local-exec): pod_gpu_memory_utilization                                          Successful
null_resource.validator (local-exec): ClusterName-InstanceId-NodeName                                     Successful
null_resource.validator (local-exec): node_gpu_memory_total                                               Successful
null_resource.validator (local-exec): node_gpu_memory_used                                                Successful
null_resource.validator (local-exec): node_gpu_power_draw                                                 Successful
null_resource.validator (local-exec): node_gpu_temperature                                                Successful
null_resource.validator (local-exec): node_gpu_utilization                                                Successful
null_resource.validator (local-exec): node_gpu_memory_utilization                                         Successful
null_resource.validator (local-exec): ClusterName-GpuDevice-InstanceId-InstanceType-NodeName              Successful
null_resource.validator (local-exec): node_gpu_memory_total                                               Successful
null_resource.validator (local-exec): node_gpu_memory_used                                                Successful
null_resource.validator (local-exec): node_gpu_power_draw                                                 Successful
null_resource.validator (local-exec): node_gpu_temperature                                                Successful
null_resource.validator (local-exec): node_gpu_utilization                                                Successful
null_resource.validator (local-exec): node_gpu_memory_utilization                                         Successful
null_resource.validator (local-exec): ClusterName-ContainerName-FullPodName-Namespace-PodName             Successful
null_resource.validator (local-exec): container_gpu_memory_total                                          Successful
null_resource.validator (local-exec): container_gpu_memory_used                                           Successful
null_resource.validator (local-exec): container_gpu_power_draw                                            Successful
null_resource.validator (local-exec): container_gpu_temperature                                           Successful
null_resource.validator (local-exec): container_gpu_utilization                                           Successful
null_resource.validator (local-exec): container_gpu_memory_utilization                                    Successful
null_resource.validator (local-exec): ClusterName-Namespace                                               Successful
null_resource.validator (local-exec): pod_gpu_memory_total                                                Successful
null_resource.validator (local-exec): pod_gpu_memory_used                                                 Successful
null_resource.validator (local-exec): pod_gpu_power_draw                                                  Successful
null_resource.validator (local-exec): pod_gpu_temperature                                                 Successful
null_resource.validator (local-exec): pod_gpu_utilization                                                 Successful
null_resource.validator (local-exec): pod_gpu_memory_utilization                                          Successful
null_resource.validator (local-exec): ClusterName-Namespace-PodName                                       Successful
null_resource.validator (local-exec): pod_gpu_memory_total                                                Successful
null_resource.validator (local-exec): pod_gpu_memory_used                                                 Successful
null_resource.validator (local-exec): pod_gpu_power_draw                                                  Successful
null_resource.validator (local-exec): pod_gpu_temperature                                                 Successful
null_resource.validator (local-exec): pod_gpu_utilization                                                 Successful
null_resource.validator (local-exec): pod_gpu_memory_utilization                                          Successful
null_resource.validator (local-exec): ClusterName-ContainerName-Namespace-PodName                         Successful
null_resource.validator (local-exec): container_gpu_memory_total                                          Successful
null_resource.validator (local-exec): container_gpu_memory_used                                           Successful
null_resource.validator (local-exec): container_gpu_power_draw                                            Successful
null_resource.validator (local-exec): container_gpu_temperature                                           Successful
null_resource.validator (local-exec): container_gpu_utilization                                           Successful
null_resource.validator (local-exec): container_gpu_memory_utilization                                    Successful
null_resource.validator (local-exec): ClusterName-FullPodName-Namespace-PodName                           Successful
null_resource.validator (local-exec): pod_gpu_memory_total                                                Successful
null_resource.validator (local-exec): pod_gpu_memory_used                                                 Successful
null_resource.validator (local-exec): pod_gpu_power_draw                                                  Successful
null_resource.validator (local-exec): pod_gpu_temperature                                                 Successful
null_resource.validator (local-exec): pod_gpu_utilization                                                 Successful
null_resource.validator (local-exec): pod_gpu_memory_utilization                                          Successful
null_resource.validator (local-exec): ClusterName                                                         Successful
null_resource.validator (local-exec): container_gpu_memory_total                                          Successful
null_resource.validator (local-exec): container_gpu_memory_used                                           Successful
null_resource.validator (local-exec): container_gpu_power_draw                                            Successful
null_resource.validator (local-exec): container_gpu_temperature                                           Successful
null_resource.validator (local-exec): container_gpu_utilization                                           Successful
null_resource.validator (local-exec): container_gpu_memory_utilization                                    Successful
null_resource.validator (local-exec): pod_gpu_memory_total                                                Successful
null_resource.validator (local-exec): pod_gpu_memory_used                                                 Successful
null_resource.validator (local-exec): pod_gpu_power_draw                                                  Successful
null_resource.validator (local-exec): pod_gpu_temperature                                                 Successful
null_resource.validator (local-exec): pod_gpu_utilization                                                 Successful
null_resource.validator (local-exec): pod_gpu_memory_utilization                                          Successful
null_resource.validator (local-exec): node_gpu_memory_total                                               Successful
null_resource.validator (local-exec): node_gpu_memory_used                                                Successful
null_resource.validator (local-exec): node_gpu_power_draw                                                 Successful
null_resource.validator (local-exec): node_gpu_temperature                                                Successful
null_resource.validator (local-exec): node_gpu_utilization                                                Successful
null_resource.validator (local-exec): node_gpu_memory_utilization                                         Successful
null_resource.validator (local-exec): emf-logs                                                            Successful
null_resource.validator (local-exec): 2024/04/03 10:53:45 ==============================
null_resource.validator (local-exec): 2024/04/03 10:53:45 >>>>>>>>>>>>>>><<<<<<<<<<<<<<<
null_resource.validator (local-exec): >>>> Finished GPU Container Insights TestSuite
null_resource.validator (local-exec): --- PASS: TestGPUSuite (188.82s)
null_resource.validator (local-exec):     --- PASS: TestGPUSuite/TestAllInSuite (188.82s)
null_resource.validator (local-exec): PASS
null_resource.validator (local-exec): ok  	github.com/aws/amazon-cloudwatch-agent-test/test/gpu	189.974s
null_resource.validator: Creation complete after 3m15s [id=6180149298917079822]

[zhihonl]

nit: TItle should be Add integ tests for Nvidia GPU in EKS

[zhihonl]

Can you link a full integration test run so we can check if it breaks existing tests

[movence]

Sure, added to the desc. There are some failures but they seem to be not related to gpu test changes.

[zhihonl]

https://github.com/aws/amazon-cloudwatch-agent/actions/runs/8542742082/job/23405557390

One of the eks_daemon tests failed, is that just flaky test?

[movence]

Seeing timeouts with:

module.windows.null_resource.fluentbit-windows (local-exec): Waiting for daemon set "fluent-bit-windows" rollout to finish: 0 of 1 updated pods are available...

which doesn't seem to be related to this PR, but I will let the change owner know about it.

[Paramadon]

looks good!