Add EKS integration test for FluentBit log emission with Entity field #427

zhihonl · 2024-11-12T05:14:42Z

Description of the issue

FluentBit has a new feature to emit CloudWatch logs with entity field which helps users navigate between their telemetry in the CloudWatch console. We want to add an integration test which helps us validate this behavior.

This PR focuses on the test scenario where FluentBit falls back to using K8sWorkload as the service name and uses default environment name.

Description of changes

Add new terraform that uses helm chart to set up the cluster
Create a worker pod that output log every second
Add test code to validate the entity fields contain the necessary values.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Test run: https://github.com/aws/amazon-cloudwatch-agent/actions/runs/11790939699/job/32939527206 (Ran after commit 861e08b)

lisguo · 2024-11-13T15:38:53Z

terraform/eks/daemon/entity/main.tf

+  role_arn = module.basic_components.role_arn
+  version  = var.k8s_version
+  enabled_cluster_log_types = [
+    "api",


do we need these? Not sure what these do

Tried running the setup without these and still works. I think these provide control plane logs from EKS under /aws/eks log group but we only care about application logs from Container Insights log group for these tests. Will remove in next commit.

lisguo · 2024-11-13T15:43:08Z

terraform/eks/daemon/entity/main.tf

+// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+// SPDX-License-Identifier: MIT
+
+module "common" {


is there no common terraform to create an eks cluster? seems like we have duplicate code here from our EKS integ tests...can we just reuse it and then apply the helm chart?

There is no common terraform to create EKS cluster. All terraform setups for each test under terraform/eks folder have their own cluster setup. I think your point is valid though, having a common code to avoid potential duplication is better but I think it's out of scope for this PR.

lisguo · 2024-11-13T15:45:20Z

test/entity/entity_test.go

+	entityServiceNameSource = "@entity.Attributes.AWS.ServiceNameSource"
+)
+
+type expectedEntity struct {


can't this just be the cloudwatch.Entity struct? Then we can have checks to ensure the KeyAttributes/Attributes are valid in the struct?

When querying the logs, the response doesn't return in cloudwatch.Entity type, it's Results [][]*ResultField which is basically arrays of strings. We could try to use cloudwatch.Entity just to hold the expected output but it would require additional logics to deference the maps just to turn them into strings so thought using a custom test struct is cleaner.

lisguo · 2024-11-13T15:45:53Z

test/entity/entity_test.go

+
+// ValidateEntity queries a given LogGroup/LogStream combination given the start and end times, and executes an
+// log query for entity attributes to ensure the entity values are correct
+func ValidateEntity(t *testing.T, logGroup, logStream string, end *time.Time, queryString string, expectedEntity expectedEntity, entityPlatformType string) {


Suggested change

func ValidateEntity(t *testing.T, logGroup, logStream string, end *time.Time, queryString string, expectedEntity expectedEntity, entityPlatformType string) {

func ValidateLogEntity(t *testing.T, logGroup, logStream string, end *time.Time, queryString string, expectedEntity expectedEntity, entityPlatformType string) {

Will fix in next commit

lisguo · 2024-11-13T15:47:01Z

test/entity/entity_test.go

+		t.Fatalf("application log group used for entity validation doesn't exsit: %s", logGroup)
+	}
+	begin := end.Add(-sleepForFlush * 2)
+	log.Printf("Start time is " + begin.String() + " and end time is " + end.String())


Let's make sure these are accurate...how are we tracking the begin time? It should be when the terraform is initiated?

Begin time is 2 time before the end point. This is the current flow:

Terraform initialize all resources(cluster, helm chart, etc)

Test starts running

Sleep 3 minutes

End time is set after 3 minutes sleep time

Begin time is 6 minutes before end time since it's doing the *2 multiplication. We could remove the multiplication but I did it just to ensure we don't somehow miss a query by barely missing the start time.

amazon-cloudwatch-agent-test/test/entity/entity_test.go

Line 74 in 4c08340

end := time.Now()

but this assumes that the cluster was created within 6 minutes before the end time right? Wondering if there is a way to start the timer at terraform init. Not sure how we are doing it for our other tests

lisguo · 2024-11-13T15:50:59Z

test/entity/entity_test.go

+	if !assert.NotZero(t, len(result)) {
+		return
+	}
+	if entityPlatformType == "EC2" {


Feel like we should have a defined constant for required KeyAttributes and Attributes for each of the platforms.

Something like

ec2RequiredKeyAttributes := map[string]struct{}{} { "Service": {}, "Environment":{}, ... }

And have some validator in a util func. Having a map[string]struct{}{} is just easier to do the checks...not sure if we have a List abstraction for this.

Going to create a validator struct that performs different logics based on platforms.

lisguo · 2024-11-13T15:52:41Z

util/awsservice/cloudwatchlogs.go

@@ -197,6 +197,47 @@ func GetLogQueryStats(logGroupName string, startTime, endTime int64, queryString
 	}
 }

+// GetLogQueryResults for the log group between start/end (in epoch seconds) for the
+// query string.
+func GetLogQueryResults(logGroupName string, startTime, endTime int64, queryString string) ([][]types.ResultField, error) {


we don't have something like this already? Something that checks for the existence of logs in a log group within a certain time period?

We have GetLogQueryStats function which does what you described but that returns the statistics of the query result which does not include log fields so we can't validate entity from that. GetLogQueryResults function is a copy of that function except that it returns results.Results

lisguo · 2024-11-13T19:43:05Z

test/entity/entity_test.go

+			agentConfigPath: filepath.Join("resources", "compass_default_log.json"),
+			podName:         "log-generator",
+			expectedEntity: expectedEntity{
+				entityType:        "Service",


nit: we could have an enum for different entity types: Service, Resource

Same for platformType: AWS::EKS, AWS::K8s, etc

Will address in the next CR where the values get re-used

zhihonl added 2 commits November 12, 2024 00:10

Add EKS integration test for FluentBit emitting entity with K8sWorkload

471fedc

Modify comment

b55ca49

zhihonl requested a review from a team as a code owner November 12, 2024 05:14

Fix linter errors

4c08340

zhihonl requested review from lisguo and varunch77 November 13, 2024 15:25

varunch77 previously approved these changes Nov 13, 2024

View reviewed changes

lisguo reviewed Nov 13, 2024

View reviewed changes

Add validator struct for different platforms entity validation

861e08b

zhihonl dismissed varunch77’s stale review via 861e08b November 13, 2024 17:16

lisguo approved these changes Nov 13, 2024

View reviewed changes

varunch77 approved these changes Nov 13, 2024

View reviewed changes

zhihonl merged commit c0aafbb into main Nov 13, 2024
2 checks passed

zhihonl deleted the eks-entity branch November 13, 2024 20:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add EKS integration test for FluentBit log emission with Entity field #427

Add EKS integration test for FluentBit log emission with Entity field #427

zhihonl commented Nov 12, 2024 •

edited

Loading

lisguo Nov 13, 2024

zhihonl Nov 13, 2024 •

edited

Loading

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

lisguo Nov 13, 2024

zhihonl Nov 13, 2024 •

edited

Loading

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

lisguo Nov 13, 2024

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

lisguo Nov 13, 2024

zhihonl Nov 13, 2024

	func ValidateEntity(t testing.T, logGroup, logStream string, end time.Time, queryString string, expectedEntity expectedEntity, entityPlatformType string) {
	func ValidateLogEntity(t testing.T, logGroup, logStream string, end time.Time, queryString string, expectedEntity expectedEntity, entityPlatformType string) {

Add EKS integration test for FluentBit log emission with Entity field #427

Add EKS integration test for FluentBit log emission with Entity field #427

Conversation

zhihonl commented Nov 12, 2024 • edited Loading

Description of the issue

Description of changes

License

Tests

Choose a reason for hiding this comment

zhihonl Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhihonl Nov 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhihonl commented Nov 12, 2024 •

edited

Loading

zhihonl Nov 13, 2024 •

edited

Loading

zhihonl Nov 13, 2024 •

edited

Loading