Retrieve instance tags at the same time to reduce number of entities in Explore related experience

[nathalapooja]

Description of the issue

Describe the problem or feature in addition to a link to the issues.
Currently we have two separate threads for reading the AutoScalingGroup tag and the service name tags (service, application, or app). This causes different entities to be created as a result which can be confusing for the customer. We should have a single thread (and retryer) to retrieve all tags needed for entity population for the Explore related experience.

Description of changes

How does this change address the problem?
Implementation

We get the ASG + Service Name + all tags on init
Get all tags on an interval together until success

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Describe what tests you have done.
Unit Tests
Manual Tests
Before enabling the instance metadata tags

Field	Value
@entity.Attributes.AWS.ServiceNameSource	ClientIamRole
@entity.Attributes.EC2.InstanceId	i-0b6fe6aab9e76873b
@entity.Attributes.PlatformType	AWS::EC2
@entity.KeyAttributes.Environment	ec2:default
@entity.KeyAttributes.Name	ec2-compass-iam-role
@entity.KeyAttributes.Type	Service
@ingestionTime	1734465217411
@log	957688854012:agent_log_group
@logStream	agent_log_stream
@message	2024-12-17T19:53:35Z D! [logagent] open file count, 1
@timestamp	1734465216354

After enabling the instance metadata tags

Field	Value
@entity.Attributes.AWS.ServiceNameSource	ResourceTags
@entity.Attributes.EC2.AutoScalingGroup	asg-compass
@entity.Attributes.EC2.InstanceId	i-0b6fe6aab9e76873b
@entity.Attributes.PlatformType	AWS::EC2
@entity.KeyAttributes.Environment	ec2:asg-compass
@entity.KeyAttributes.Name	poojardy_test_app
@entity.KeyAttributes.Type	Service
@ingestionTime	1734465495685
@log	957688854012:agent_log_group
@logStream	agent_log_stream
@message	2024-12-17T19:58:14Z D! [logagent] open file count, 1
@timestamp	1734465495195

Agents unlimited retried until successful retrieval of tags based of BackOffSleepArray. Stopped retying after first success
BackoffSleepArray = []time.Duration{0, 1 * time.Minute, 1 * time.Minute, 3 * time.Minute, 3 * time.Minute, 3 * time.Minute, 10 * time.Minute}

[ec2-user@ip-172-31-6-118 ~]$ sudo cat /opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log | grep "Scraping"
2024-12-17T19:16:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:16:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:17:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:18:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:21:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:24:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:27:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:37:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:47:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:57:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
[ec2-user@ip-172-31-6-118 ~]$ date
Tue Dec 17 20:21:47 UTC 2024

I enabled the instance tags at 19:53 and You can see above there is no retry after 19:57 until 20:21 current time.

2024-12-17T19:57:39Z I! {"caller":"entitystore/serviceprovider.go:248","msg":"Scraping IMDS for service name and AutoScalingGroup","kind":"extension","name":"entitystore"}
2024-12-17T19:57:39Z I! {"caller":"entitystore/serviceprovider.go:261","msg":"Service name retrieved through IMDS","kind":"extension","name":"entitystore"}
2024-12-17T19:57:39Z I! {"caller":"entitystore/serviceprovider.go:274","msg":"AutoScalingGroup retrieved through IMDS","kind":"extension","name":"entitystore"}
2024-12-17T19:57:40Z D! [logagent] open file count, 1
2024-12-17T19:57:40Z D! [outputs.cloudwatchlogs] Pusher published 13 log events to group: agent_log_group stream: agent_log_stream with size 4 KB in 9.93404ms.

Requirements

Before commit the code, please do the following steps.

1. Run make fmt and make fmt-sh
2. Run make lint

[lisguo]

Do we need this strings.Contains if statement? We end up checking if the instance tag exists anyways in the next line. Seems excessive.

[nathalapooja]

We only make a call if it exists in the list of tags

[lisguo]

sure, but in the end we would error out from s.metadataProvider.InstanceTagValue if it doesn't exist right?

[nathalapooja]

Yes, but its a safety check to avoid additional calls

[lisguo]

Approving since we are being consistent with existing code -- but worth revisiting removing the ASG char limit and checking for strings.Contains for the entire tags dump