Detect AWS region from instance metadata

[mikelorant]

When starting discovery, receive the following error:

operation error Kafka: ListClusters, failed to resolve service endpoint, an AWS region is required, but was not found

This can be resolved by setting the AWS_REGION environmental variable but this can also be determined from the instance metadata so it would not be necessary to configure anything (as long as the MSK clusters in the same region).

[joshm91]

Hey Mike,

Thanks for this! I'm guessing this was running the binary directly on an EC2 instance? It's not something that was caught since we generally only use it in the Docker image on an EKS cluster which tends to set the required environment variables.

It looks like an easy fix so let me just test it some more and I'll push the new version.

[mikelorant]

I am running it in a pod within our Kops built Kubernetes cluster.

It was just the sidecar container of the Prometheus Helm chart. Nothing special at all.

[joshm91]

Got it, thanks!

When you get a chance, can you try the Docker image: statsbomb/prometheus-msk-discovery:ec2region and let me know if it works as expected.

[imriz]

I just encountered the same issue, and statsbomb/prometheus-msk-discovery:ec2region solved it for me.
As a suggestion - maybe add an option to explicitly set it via CLI (although I assume the standard AWS env var will also work?).
Another one, more important, is to allow passing a list of regions, to scan them all.

[joshm91]

@imriz Thanks for confirming! I'll create a proper release with the fix today.

As for the list of regions I've created a dedicated feature request for it (#10) but I suspect it'd require a bit of rewrite to implement so I can't promise it'll be done any time soon.

Allowing specifying a single region as a parameter shouldn't be too difficult (#11) but in the meantime I'd imagine you could just set the AWS_REGION environment variable to the desired region and things should work.

[joshm91]

@mikelorant @imriz The image statsbomb/prometheus-msk-discovery:0.3.0 now contains the EC2 IMDS fallback as well as the ability to specify a region with the -region flag.

Let me know if you have any problems!