Adding S3 bucket policies to lock down unsecure access

[jpl-btlunsfo]

Purpose

• Adds s3 bucket policy to block unsecure access on the various S3 buckets created by the unity-sps terraform

Proposed Changes

• ADD aws_s3_bucket_policy entries for the 4 buckets that are created (one in terraform-unity-sps-airflow, three in terraform-unity-sps-initiators)

Issues

• [New Feature]: Enforce SSL access on SPS S3 buckets #231

Testing

• Tested in personal deployment; deployed without issues...
• Unsure of how to test the initiators buckets.

[LucaCinquini]

Hi @jpl-btlunsfo : the changes to the code look ok, but can you try to run some DAGs to make sure they still work?

[jpl-btlunsfo]

@LucaCinquini no issues running the cwl dag as a test, and while my attempted run of the sbg step by step dag failed I don't think that was related to this- the airflow-related S3 bucket is for the airflow logs, and the (failed) logs are present/were uploaded without issue.

[LucaCinquini]

@jpl-btlunsfo : I am getting this error when deploying Airflow with this branch. Deployment of EKS and Karpenter were successfull:

module.unity-sps-ogc-processes-api.kubernetes_deployment.ogc_processes_api: Still creating... [3m20s elapsed]
module.unity-sps-ogc-processes-api.kubernetes_deployment.ogc_processes_api: Still creating... [3m30s elapsed]
module.unity-sps-ogc-processes-api.kubernetes_deployment.ogc_processes_api: Creation complete after 3m38s [id=sps/ogc-processes-api]
╷
│ Error: creating S3 Bucket (unity-luca-1-dev-sps-airflowlogs): operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: P863FADYWY73YADB, HostID: A1gMq0R27T/Xuxefmmw4qTqzyeDXTzLWsUJw7Ee1yYrywbolP9ZFgtje/+hjz44s07Z3jWs69DY=, api error TooManyBuckets: You have attempted to create more buckets than allowed
│
│ with module.unity-sps-airflow.aws_s3_bucket.airflow_logs,
│ on modules/terraform-unity-sps-airflow/main.tf line 117, in resource "aws_s3_bucket" "airflow_logs":
│ 117: resource "aws_s3_bucket" "airflow_logs" {
│
╵
╷
│ Error: creating S3 Bucket (unity-luca-1-dev-sps-config): operation error S3: CreateBucket, https response error StatusCode: 400, RequestID: AD9RZ779H94SBQQ7, HostID: bGJYw353obYeM82RX6DK/vVDuAJLUOlgUiYuKzMpIr20LMcOVQRQsG+hKtT/1mprRtgTtzst0RM=, api error TooManyBuckets: You have attempted to create more buckets than allowed
│
│ with module.unity-sps-initiators.aws_s3_bucket.config,
│ on modules/terraform-unity-sps-initiators/main.tf line 21, in resource "aws_s3_bucket" "config":
│ 21: resource "aws_s3_bucket" "config" {
│

[jpl-btlunsfo]

TooManyBuckets: You have attempted to create more buckets than allowed

I don't believe that's due to the S3 bucket policy change- that looks like we've made too many buckets. Maybe @galenatjpl has seen this before?

[LucaCinquini]

I don't have that problem if I use the code in develop, though...

[LucaCinquini]

I don't have that problem if I use the code in develop, though. I will try again later.

[galenatjpl]

@jpl-btlunsfo @LucaCinquini , yes it appears we are hitting a limit. I'm cleaning up some buckets now

[LucaCinquini]

@jpl-btlunsfo : the deployment with this branch works for me if this line:

unity-sps/terraform-unity/modules/terraform-unity-sps-initiators/main.tf

Line 37 in fa5c18d

bucket = format(local.resource_name_prefix, each.value)

is changed from "each.value" to "each.key"

[jpl-btlunsfo]

Ah, yeah it looks like I'd left that format in when it wasn't necessary. I've reworked that resource a bit, since it doesn't really need key/values in that for_each, and retested. Could you give it a retest too @LucaCinquini ?

[LucaCinquini]

After the latest changes, the deployment worked for me, please go ahead and merge into develop.