Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add IMDS fallback with DescribeInstances. #1139

Merged
merged 2 commits into from
Apr 17, 2024
Merged

Add IMDS fallback with DescribeInstances. #1139

merged 2 commits into from
Apr 17, 2024

Conversation

jefchien
Copy link
Contributor

@jefchien jefchien commented Apr 15, 2024 •
edited
Loading

Description of the issue

Related to #1101.

Instance Metadata Service (IMDS) provides key information about the local EC2 instance including region and instance ID. The CloudWatch Agent relies on IMDS to retrieve those fields, which it uses as metric dimensions. As a security best practice, EKS recommends that the hop limit on worker nodes is set to 1. This effectively disables IMDS within containers as any response for a call made within a container will require at least 2 hops.

In the cases where IMDS is not available or disabled, the agent needs an additional fallback mechanism to retrieve the metadata without IMDS. For EC2 instances, the default AWS hostname follows a fixed format and will be prefixed with either the private IPv4 address (ip-10-24-34-0.ec2.internal) or the instance ID (i-0123456789abcdef.us-west-2.compute.internal) based on the hostname type configured on the instance. In these cases, DescribeInstances can be used to get the instance metadata instead.

Description of changes

Adds a fallback metadata provider that parses the hostname to determine an EC2 filter and makes a call to DescribeInstances with that filter. Consolidates the metadata provider functions into a single package.

License

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Tests

Added unit tests and built/installed the agent on an EC2 host with IMDS disabled. Configured common-config.toml to point to shared credential file.

fmt.Print to stdout isn't being logged for some reason. The IMDS logging is from the AWS SDK.

Without DescribeInstances permissions

> sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -c file:test.json -s
****** processing amazon-cloudwatch-agent ******
2024/04/17 15:35:28 I! WARN: failed to get session token, falling back to IMDSv1: 403 Forbidden: Forbidden
	status code: 403, request id: 
caused by: EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
2024/04/17 15:35:29 E! Please check if you can access the metadata service. For example, on linux, run 'wget -q -O - http://169.254.169.254/latest/meta-data/instance-id && echo'

With DescribeInstances permissions

> sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -c file:test.json -s
****** processing amazon-cloudwatch-agent ******
2024/04/17 16:23:46 I! WARN: failed to get session token, falling back to IMDSv1: 403 Forbidden: Forbidden
	status code: 403, request id: 
caused by: EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
Got Home directory: /root I! Set home dir Linux: /root I! SDKRegionWithCredsMap region: I! Trying to detect region from ec2 D! [EC2] Found active network interface I! imds retry client will retry 1 times D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>403 - Forbidden</title> </head> <body> <h1>403 - Forbidden</h1> </body> </html> status code: 403, request id: D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>403 - Forbidden</title> </head> <body> <h1>403 - Forbidden</h1> </body> </html> status code: 403, request id: D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>403 - Forbidden</title> </head> <body> <h1>403 - Forbidden</h1> </body> </html> status code: 403, request id: D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request <?xml version="1.0" encoding="iso-8859-1"?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <title>403 - Forbidden</title> </head> <body> <h1>403 - Forbidden</h1> </body> </html> status code: 403, request id: Successfully fetched the config and saved in /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_test2.json.tmp
Start configuration validation...
2024/04/17 16:23:46 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d/file_test2.json.tmp ...
2024/04/17 16:23:46 I! Valid Json input schema.
2024/04/17 16:23:47 I! WARN: failed to get session token, falling back to IMDSv1: 403 Forbidden: Forbidden
	status code: 403, request id: 
caused by: EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
2024/04/17 16:23:47 D! ec2tagger processor required because append_dimensions is set
2024/04/17 16:23:47 D! delta processor required because metrics with diskio or net are set
2024/04/17 16:23:47 D! ec2tagger processor required because append_dimensions is set
2024/04/17 16:23:47 Configuration validation first phase succeeded
I! Detecting run_as_user...
D! [EC2] Found active network interface
I! imds retry client will retry 1 times
D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
D! should retry true for imds error : EC2MetadataError: failed to make EC2Metadata request
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>403 - Forbidden</title>
 </head>
 <body>
  <h1>403 - Forbidden</h1>
 </body>
</html>

	status code: 403, request id: 
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent -schematest -config /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
Configuration validation second phase succeeded
Configuration validation succeeded

Requirements

Before commit the code, please do the following steps.

  1. Run make fmt and make fmt-sh
  2. Run make lint

@jefchien jefchien requested a review from a team as a code owner April 15, 2024 20:22
okankoAMZ
okankoAMZ approved these changes Apr 17, 2024
View reviewed changes
Copy link
Contributor

@okankoAMZ okankoAMZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Left few questions

internal/metadata/ec2/non_imds.go Show resolved Hide resolved
plugins/processors/ec2tagger/ec2tagger.go Outdated Show resolved Hide resolved
@jefchien jefchien merged commit 8232511 into feature-imds-fallback Apr 17, 2024
@jefchien jefchien deleted the imds-fallback branch April 17, 2024 16:42
@jefchien jefchien mentioned this pull request Apr 18, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@okankoAMZ okankoAMZ okankoAMZ approved these changes

@sethAmazon sethAmazon sethAmazon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@jefchien @sethAmazon @okankoAMZ