Can't find external ID upon using Grafana Assume Role

[katebrenner]

What happened:
External ID is not displayed while creating grafana-athena-datasource when the Authentication Provider is selected as Grafana Assume Role. Tt seems like this is only a problem with new Athena datasources that enable Grafana Assume role. I tried it with a new Cloudwatch datasource in the customer's account and that one works fine. More info in the linked issue and comments!

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Screenshots

Anything else we need to know?:

Environment:

• Grafana version:
• Plugin version:
• OS Grafana is installed on:
• User OS & Browser:
• Others:

[sarahzinger]

Notes:

In addition to recreating this in Grafana Cloud, I can now reliably recreate this locally by doing the following:
set custom.ini:
allowed_auth_providers = keys,grafana_assume_role
external_id= anything

and then ensure assume_role_enabled is not set.

Then create a new athena datasource, select grafana assume role, and you'll get a 500 with the error [plugin.downstreamError] client: failed to call resources: failed to receive call resource response: rpc error: code = Unknown desc = attempting to use an auth type that is not allowed: \"default\": Failed to create client

That error is coming from grafana-aws-sdk for go in sessions.go

It seems like maybe we made a bad assumption that assume_role_enabled was true by default in cloud? Or maybe something recently changed to unset that default?

In addition we seem to be forcing a backup auth type into the "default" which doesn't make sense because that is not an allowed auth type in cloud.

I don't know that it makes sense to me that this only happens in athena and not cloudwatch, maybe Athena has a default query that selects SDK default by default or as a fallback?

Going to keep debugging

[sarahzinger]

Ok I think the reason this happens in Athena and not Cloudwatch is because Athena uses the sqlds package which first tries to connect to the database before calling the sqlds New Datasource function that adds the custom routes of external Id

See here:
https://github.com/grafana/grafana-aws-sdk/blob/ba2e2a85bde3c3c2ab30b70759cb346f4107615f/pkg/awsds/asyncDatasource.go#L79-L90

And here:
https://github.com/grafana/sqlds/blob/71090eb4d9b6af7c745e2c238131e8d107e63191/datasource.go#L84-L102

This feels like a not obvious bug to fix. We have a couple of different possibilities that I can think of:

• we could try to add some kind of override to set a different default auth type besides "default" (arguably default should have never been our default since that is not supported in grafana cloud). The challenge here, is I think that changing the default auth type at this point would probably cause unwanted breaking changes for many people, so I don't think this would be an option? Could be wrong on that though?
• we could add some code to allow for custom routes that are set before a connection to the database is made. Seems like other sqlds plugins could have this need? My only hesitation is it feels like adding more complexity to an already complex system? But I could see it making some sense.
• alternatively we could try to reduce the tight coupling here of athena on sqlds. The state of debugging these issues is surprirsingly difficult. I had to install sqlds, grafana-aws-sdk, grafana-aws-sdk-react locally on my machine to go through the basic flow of information of loading config the config editor and hitting a resource endpoint. Maybe the existing abstractions of sqlds do not help athena as much as we'd like?

In my ideal world athena:

• is responsible for and owns it's own resource handling, but uses some kind of helpers from sqlds or sqlutils to create common ones.
• then if we want to add to it for athena-specific instances we can.

But all of this will take time to work out a plan.

In the mean time there is a work around for this particular bug of a feature that's still in private preview for users. They need to save the (broken/empty) datasource first, then refresh the page in their browser, and then try to see the external id. Doing so will set an auth type of grafana_assume_role rather than "default", and it should work as expected. It's a wonky UI but hopefully not too burdensome as it's a one time annoyance on set up.

I think the fact that this is not an easy problem to fix, is probably reflective of some bigger architectural challenges, and we have a couple of bigger projects coming up: Multi-tenancy and upgrading the new aws-sdk for go v2 that I think will end up forcing us to change a lot of this architecture, and hopefully make it easier to solve this. I think so long as the work around works, it might be better to hold off figuring out how to rewrite this until then, so we do not duplicate efforts. Going to put this in the backlog for now.