Skip to content

Commit

Permalink
Addendums to readme for s3 migration (#18)
Browse files Browse the repository at this point in the history
* Addendums to readme for s3 migration

Specifically, this covers additional details
that might prove useful to those try to migrate
from template deploy to live-1.

I have put this here since the user guide already
links to this page from [here](https://user-guide.cloud-platform.service.justice.gov.uk/tasks.html#migrating-an-s3-bucket)
  • Loading branch information
jsugarman authored Oct 24, 2019
1 parent 48f9d0b commit 7bf4975
Showing 1 changed file with 52 additions and 2 deletions.
54 changes: 52 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,41 @@ Some of the inputs are tags. All infrastructure resources need to be tagged acco

The `user_policy` input is useful when migrating data from existing bucket(s). For commands like `s3 ls` or `s3 sync` to work across accounts, a policy granting access must be set in 2 places: the *source bucket* and the *destination user*

Example for the source bucket:

### Source bucket policy

The source bucket must permit the destination s3 IAM user to "read" from its bucket explcitly.

Example to retrieve destination IAM user for use in source bucket policy. _requires [jq - commandline JSON processer](https://stedolan.github.io/jq/)_

```bash
# retrieve destination s3 user ARN

# retrieve live-1 namespace's s3 credentials
$ kubectl -n my-namespace get secret my-s3-secrets -o json | jq -r '.data[] | @base64d'
=>
<access_key_id>
<bucket_arn>
<bucket_name>
<secret_access_key>

# retrieve IAM user details using credentials
$ unset AWS_PROFILE; AWS_ACCESS_KEY_ID=<access_key_id> AWS_SECRET_ACCESS_KEY=<secret_access_key> aws sts get-caller-identity

# Alternative single call in bash
$ unset AWS_PROFILE; read K a n S <<<$(kubectl -n my-namespace get secret my-s3-secrets -o json | jq -r '.data[] | @base64d') ; AWS_ACCESS_KEY_ID=$K AWS_SECRET_ACCESS_KEY=$S aws sts get-caller-identity
```

You should get output similar to below:
```json
{
"UserId": "<userid>",
"Account": "<accountid>",
"Arn": "arn:aws:iam::<accountid>:user/system/s3-bucket-user/<team>/<random-s3-bucket-username>"
}
```

Example for the source bucket (using retrieved ARN from above):

```
{
Expand All @@ -94,8 +128,10 @@ Example for the source bucket:
}
```

Note the bucket being listed twice, this is needed not a typo.
Note the bucket being listed twice, this is needed not a typo - the first is for the bucket itself, second for objects within it.


### Destination IAM user policy
Example for the destination IAM user created by this module:

```
Expand Down Expand Up @@ -130,3 +166,17 @@ Example for the destination IAM user created by this module:
}
EOF
```

### Synchronization

Once configured the following, executed within a relevantly provisioned pod in the destination namespace, will add new, update existing and delete objects (not in source).

```bash
aws s3 sync --delete \
s3://source_bucket_name \
s3://destination_bucket_name \
--source-region source_region \
--region destination_region
```

For an example of a pod with a custom CLI that wraps s3 sync you can see the [cccd-migrator](https://github.com/ministryofjustice/cccd-migrator)

0 comments on commit 7bf4975

Please sign in to comment.