diff --git a/README.md b/README.md index 2fc3c9f..e2faa2c 100644 --- a/README.md +++ b/README.md @@ -26,42 +26,110 @@ module "s3" { ### Migrate from existing buckets - +You can use a combination of the [Cloud Platform IRSA module](https://github.com/ministryofjustice/cloud-platform-terraform-irsa) and [Service pod module](https://github.com/ministryofjustice/cloud-platform-terraform-service-pod) to access your source bucket using the AWS CLI. -The `user_policy` input is useful when migrating data from existing bucket(s). For commands like `s3 ls` or `s3 sync` to work across accounts, a policy granting access must be set in 2 places: the *source bucket* and the *destination user* +#### IRSA and Service Pod example configuration +In the [cloud-platform-environments](https://github.com/ministryofjustice/cloud-platform-environments) repository, within your namespace which contains your destination s3 bucket configuration, add the following terraform, substituting values as necessary: -#### Source bucket policy +``` +module "cross_irsa" { + source = "github.com/ministryofjustice/cloud-platform-terraform-irsa?ref=[latest-release-here]" + business_unit = var.business_unit + application = var.application + eks_cluster_name = var.eks_cluster_name + namespace = var.namespace + service_account_name = "${var.namespace}-cross-service" + is_production = var.is_production + team_name = var.team_name + environment_name = var.environment + infrastructure_support = var.infrastructure_support + role_policy_arns = { s3 = aws_iam_policy.s3_migrate_policy.arn } +} -The source bucket must permit the destination s3 IAM user to "read" from its bucket explcitly. +data "aws_iam_policy_document" "s3_migrate_policy" { + # List & location for source & destination S3 bucket. + statement { + actions = [ + "s3:ListBucket", + "s3:GetBucketLocation" + ] + resources = [ + module.s3_bucket.bucket_arn, + "arn:aws:s3:::[source-bucket-name]" + ] + } + # Permissions on source S3 bucket contents. + statement { + actions = [ + "s3:GetObject", + "s3:GetObjectVersion", + "s3:GetObjectTagging" + ] + resources = [ "arn:aws:s3:::[source-bucket-name]/*" ] # take note of trailing /* here + } + # Permissions on destination S3 bucket contents. + statement { + actions = [ + "s3:PutObject", + "s3:PutObjectTagging", + "s3:GetObject", + "s3:DeleteObject" + ] + resources = [ "${module.s3_bucket.bucket_arn}/*" ] + } +} + +resource "aws_iam_policy" "s3_migrate_policy" { + name = "s3_migrate_policy" + policy = data.aws_iam_policy_document.s3_migrate_policy.json + + tags = { + business-unit = var.business_unit + application = var.application + is-production = var.is_production + environment-name = var.environment + owner = var.team_name + infrastructure-support = var.infrastructure_support + } +} -Example to retrieve destination IAM user for use in source bucket policy. _requires [jq - commandline JSON processer](https://stedolan.github.io/jq/)_ +# store irsa rolearn in k8s secret for retrieving to provide within source bucket policy +resource "kubernetes_secret" "cross_irsa" { + metadata { + name = "cross-irsa-output" + namespace = var.namespace + } + data = { + role = module.cross_irsa.role_name + rolearn = module.cross_irsa.role_arn + serviceaccount = module.cross_irsa.service_account.name + } +} -```bash -# retrieve destination s3 user ARN +# set up the service pod +module "cross_service_pod" { + source = "github.com/ministryofjustice/cloud-platform-terraform-service-pod?ref=[latest-release-here]" + namespace = var.namespace + service_account_name = module.cross_irsa.service_account.name +} +``` -# retrieve live-1 namespace's s3 credentials -$ kubectl -n my-namespace get secret my-s3-secrets -o json | jq -r '.data[] | @base64d' -=> - - - - -# retrieve IAM user details using credentials -$ unset AWS_PROFILE; AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= aws sts get-caller-identity +#### Source bucket policy + +The source bucket must permit your IRSA role to "read" from its bucket explicitly. -# Alternative single call in bash -$ unset AWS_PROFILE; read K a n S <<<$(kubectl -n my-namespace get secret my-s3-secrets -o json | jq -r '.data[] | @base64d') ; AWS_ACCESS_KEY_ID=$K AWS_SECRET_ACCESS_KEY=$S aws sts get-caller-identity +First, retrieve the IRSA rolearn using cloud-platform CLI and [jq](https://jqlang.github.io/jq/) + +``` +cloud-platform decode-secret -s cross-irsa-output | jq -r '.data.rolearn' ``` You should get output similar to below: -```json -{ -"UserId": "", -"Account": "", -"Arn": "arn:aws:iam:::user/system/s3-bucket-user//" -} + +``` +arn:aws:iam::754256621582:role/cloud-platform-irsa-randomstring1234 ``` Example for the source bucket (using retrieved ARN from above): @@ -78,7 +146,7 @@ Example for the source bucket (using retrieved ARN from above): "s3:GetObject" ], "Principal": { - "AWS": "arn:aws:iam:::user/system/s3-bucket-user//s3-bucket-user-random" + "AWS": "arn:aws:iam::754256621582:role/cloud-platform-irsa-randomstring1234" }, "Resource": [ "arn:aws:s3:::source-bucket", @@ -91,47 +159,13 @@ Example for the source bucket (using retrieved ARN from above): Note the bucket being listed twice, this is needed not a typo - the first is for the bucket itself, second for objects within it. -#### Destination IAM user policy -Example for the destination IAM user created by this module: - -``` - user_policy = < env: - - name: AWS_ACCESS_KEY_ID - value: - - name: AWS_SECRET_ACCESS_KEY - value: - name: S3_BUCKET value: resources: {} @@ -207,6 +236,14 @@ spec: storage: 50Gi ``` +For further guidance on using IRSA, for example accessing AWS buckets in different accounts, see the following links: + +[Use IAM Roles for service accounts to access resources in a different AWS account](https://user-guide.cloud-platform.service.justice.gov.uk/documentation/other-topics/access-cross-aws-resources-irsa-eks.html) + +[Accessing AWS APIs and resources from your namespace](https://user-guide.cloud-platform.service.justice.gov.uk/documentation/other-topics/accessing-aws-apis-and-resources-from-your-namespace.html#accessing-aws-apis-and-resources-from-your-namespace) + +[Cloud Platform service pod for AWS CLI access]https://user-guide.cloud-platform.service.justice.gov.uk/documentation/other-topics/cloud-platform-service-pod.html) + See the [examples/](examples/) folder for more information.