DynamoDB migration is unable to read credentials. #122
Comments
|
FYI - my credentials are set in: I also tried exporting to environment variables As well as specifying in config.yaml. When I specify in config.yaml, it gives a different error: |
|
This seems to be requiring instance profile rather than using config file. I don't have permissions to change the instance profile. I'm not sure how to change the code to get it to use the default chain, but I think that would do the trick. As it would then walk through different/common methods of providing auths. |
|
@pdbossman Thank you for the detailed report. I confirm that I reproduced the issue when the credentials are provided via However, I could not reproduce your problem when the credentials are provided in the |
|
Hi Julien, The generated credentials expire. I was providing the access key and secret access key from the generated security credentials, but it's clear to me now they cannot be used in that way and are tied to the token. So when I normally run, and on the previous version of scylla-migrator, the source credentials are completely commented out. After running gimme-aws-creds, I would run aws configure, and the access key and secret key were pre-filled in from what gimme-aws-creds created, I only really had to run it to have it set the region. Then I didn't need to provide anything in the yaml file at all from the source except type, table name, and scanSegments. Basically, I need this to work without providing credentials in the yaml file at all. If we need to have a quick meeting Monday, let me know. |
|
Actually, to be clear - I think if you just fix the aws/credentials file usage for worker, you'll have solved the problem. |
|
@pdbossman I was able to use the AWS profile credentials with the following change: https://github.com/julienrf/scylla-migrator/tree/aws-credentials Could please let me know if that fixes your issue? |
It does! Thank you! |
|
so we need it configurable |
|
fwiw for our access @pdbossman we use assumed role, so we will need support for something like this: |
|
next step would be to make it configurable, so basically it will either go down hierarchy until it finds credentials |
|
com.amazonaws.auth.InstanceProfileCredentialsProvider |
Attempted to migrate from DynamoDB
I ran aws configure, and from the master and workers, I am able to list DynamoDB tables:
Source dynamodb:
aws dynamodb list-tables
{
"TableNames": [
"monitoring",
"redacted-table-name-here",
"tfstate-locks"
]
}
target scylla (I have a /etc/hosts assigning scylla hostname to proper ip):
aws dynamodb list-tables --endpoint-url "http://scylla:8000"
{
"TableNames": [
"redacted-table-name-here"
]
}
When I run spark-submit, it's hung looking for security credentials.
@hopugop @tarzanek @erezvelan
The text was updated successfully, but these errors were encountered: