Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cleanup config connector historically used to provision gcp buckets #3778

Merged
merged 5 commits into from
Mar 6, 2024
Merged

Cleanup config connector historically used to provision gcp buckets #3778

merged 5 commits into from
Mar 6, 2024

Conversation

consideRatio
Copy link
Contributor

@consideRatio consideRatio commented Mar 6, 2024 •
edited
Loading

The pangeo-hubs part was partially done already via #1160, but leaving it activated so it looked like this in pangeo-hubs cluster:

kubectl get IAMServiceAccount -A
NAMESPACE   NAME              AGE      READY   STATUS      STATUS AGE
prod        prod-user-sa      2y144d   False   Unmanaged   270d
staging     staging-user-sa   2y144d   False   Unmanaged   270d

kubectl get IAMPolicyMember -A 
NAMESPACE   NAME                        AGE      READY   STATUS      STATUS AGE
prod        sa-requester-pays-binding   2y144d   False   Unmanaged   270d
prod        scratch-bucket-binding      2y144d   False   Unmanaged   270d
staging     sa-requester-pays-binding   2y144d   False   Unmanaged   270d
staging     scratch-bucket-binding      2y144d   False   Unmanaged   270d

Changes

  • transitions the unused empty scratch buckets in 2i2c cluster for the dask-staging and ohw hubs to terraform
  • transitions specification of environment variables for pangeo-hubs
  • cleanups pangeo-hubs unused legacy bucket stuff under cloudResources
  • cleanup unused stuff related to GCP resources
  • update documentation in cluster design related to use of config connector

References

@consideRatio
2i2c, dask-staging and ohw: transition to terraform provided buckets
371e411
@consideRatio
2i2c, ohw: cleanup redundant egress config
030f871 
This is redundant since z2jh now provides egressAllowRules separate from
the `egress` list of additional rules, so we don't end up overwriting
anything by changing it.

The `egressAllowRules` include DNS and non-private IPs for port 80 and
443 communication to other places.
@consideRatio consideRatio requested a review from a team as a code owner March 6, 2024 09:31
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2024 •
edited
Loading

Merging this PR will trigger the following deployment actions.

Support and Staging deployments

Cloud Provider Cluster Name Upgrade Support? Reason for Support Redeploy Upgrade Staging? Reason for Staging Redeploy
gcp hhmi No Yes Core infrastructure has been modified
aws jupyter-meets-the-earth No Yes Core infrastructure has been modified
aws smithsonian No Yes Core infrastructure has been modified
gcp pangeo-hubs No Yes Core infrastructure has been modified
gcp awi-ciroh No Yes Core infrastructure has been modified
aws nasa-veda No Yes Core infrastructure has been modified
gcp meom-ige No Yes Core infrastructure has been modified
aws victor No Yes Core infrastructure has been modified
aws ubc-eoas No Yes Core infrastructure has been modified
aws earthscope No Yes Core infrastructure has been modified
aws catalystproject-africa No Yes Core infrastructure has been modified
gcp catalystproject-latam No Yes Core infrastructure has been modified
kubeconfig utoronto No Yes Core infrastructure has been modified
gcp qcl No Yes Core infrastructure has been modified
gcp 2i2c No Yes Core infrastructure has been modified
aws gridsst No Yes Core infrastructure has been modified
gcp 2i2c-uk No Yes Core infrastructure has been modified
gcp leap No Yes Core infrastructure has been modified
aws 2i2c-aws-us No Yes Core infrastructure has been modified
aws openscapes No Yes Core infrastructure has been modified
gcp cloudbank No Yes Core infrastructure has been modified
aws nasa-esdis No Yes Core infrastructure has been modified
gcp linked-earth No Yes Core infrastructure has been modified
aws nasa-cryo No Yes Core infrastructure has been modified
aws nasa-ghg No Yes Core infrastructure has been modified
aws opensci No Yes Following prod hubs require redeploy: sciencecore

Production deployments

Cloud Provider Cluster Name Hub Name Reason for Redeploy
gcp hhmi prod Core infrastructure has been modified
gcp hhmi spyglass Core infrastructure has been modified
aws jupyter-meets-the-earth prod Core infrastructure has been modified
aws smithsonian prod Core infrastructure has been modified
gcp pangeo-hubs prod Core infrastructure has been modified
gcp pangeo-hubs coessing Core infrastructure has been modified
gcp awi-ciroh prod Core infrastructure has been modified
aws nasa-veda prod Core infrastructure has been modified
gcp meom-ige prod Core infrastructure has been modified
aws victor prod Core infrastructure has been modified
aws ubc-eoas prod Core infrastructure has been modified
aws earthscope prod Core infrastructure has been modified
aws catalystproject-africa nm-aist Core infrastructure has been modified
aws catalystproject-africa must Core infrastructure has been modified
aws catalystproject-africa uvri Core infrastructure has been modified
gcp catalystproject-latam unitefa-conicet Core infrastructure has been modified
gcp catalystproject-latam cicada Core infrastructure has been modified
gcp catalystproject-latam gita Core infrastructure has been modified
kubeconfig utoronto prod Core infrastructure has been modified
kubeconfig utoronto r-prod Core infrastructure has been modified
gcp qcl prod Core infrastructure has been modified
gcp 2i2c imagebuilding-demo Core infrastructure has been modified
gcp 2i2c demo Core infrastructure has been modified
gcp 2i2c ohw Core infrastructure has been modified
gcp 2i2c aup Core infrastructure has been modified
gcp 2i2c temple Core infrastructure has been modified
gcp 2i2c ucmerced Core infrastructure has been modified
gcp 2i2c climatematch Core infrastructure has been modified
gcp 2i2c mtu Core infrastructure has been modified
gcp 2i2c tufts Core infrastructure has been modified
aws gridsst prod Core infrastructure has been modified
gcp 2i2c-uk lis Core infrastructure has been modified
gcp leap prod Core infrastructure has been modified
aws 2i2c-aws-us showcase Core infrastructure has been modified
aws 2i2c-aws-us ncar-cisl Core infrastructure has been modified
aws 2i2c-aws-us go-bgc Core infrastructure has been modified
aws 2i2c-aws-us itcoocean Core infrastructure has been modified
aws 2i2c-aws-us cosmicds Core infrastructure has been modified
aws openscapes prod Core infrastructure has been modified
gcp cloudbank bcc Core infrastructure has been modified
gcp cloudbank ccsf Core infrastructure has been modified
gcp cloudbank csm Core infrastructure has been modified
gcp cloudbank dvc Core infrastructure has been modified
gcp cloudbank elcamino Core infrastructure has been modified
gcp cloudbank evc Core infrastructure has been modified
gcp cloudbank glendale Core infrastructure has been modified
gcp cloudbank howard Core infrastructure has been modified
gcp cloudbank miracosta Core infrastructure has been modified
gcp cloudbank skyline Core infrastructure has been modified
gcp cloudbank demo Core infrastructure has been modified
gcp cloudbank fresno Core infrastructure has been modified
gcp cloudbank humboldt Core infrastructure has been modified
gcp cloudbank laney Core infrastructure has been modified
gcp cloudbank sbcc Core infrastructure has been modified
gcp cloudbank sbcc-dev Core infrastructure has been modified
gcp cloudbank elac Core infrastructure has been modified
gcp cloudbank lacc Core infrastructure has been modified
gcp cloudbank lamission Core infrastructure has been modified
gcp cloudbank mills Core infrastructure has been modified
gcp cloudbank mission Core infrastructure has been modified
gcp cloudbank norco Core infrastructure has been modified
gcp cloudbank palomar Core infrastructure has been modified
gcp cloudbank pasadena Core infrastructure has been modified
gcp cloudbank sjcc Core infrastructure has been modified
gcp cloudbank sacramento Core infrastructure has been modified
gcp cloudbank srjc Core infrastructure has been modified
gcp cloudbank saddleback Core infrastructure has been modified
gcp cloudbank santiago Core infrastructure has been modified
gcp cloudbank sjsu Core infrastructure has been modified
gcp cloudbank sierra Core infrastructure has been modified
gcp cloudbank tuskegee Core infrastructure has been modified
gcp cloudbank wlac Core infrastructure has been modified
gcp cloudbank csulb Core infrastructure has been modified
gcp cloudbank csum Core infrastructure has been modified
aws nasa-esdis prod Core infrastructure has been modified
gcp linked-earth prod Core infrastructure has been modified
aws opensci sciencecore Core infrastructure has been modified
aws nasa-cryo prod Core infrastructure has been modified
aws nasa-ghg prod Core infrastructure has been modified

@consideRatio consideRatio marked this pull request as draft March 6, 2024 09:48
@consideRatio consideRatio changed the title 2i2c, dask-staging and ohw: transition to terraform provided buckets Cleanup legacy bucket infrastructure and transition remaining scratch buckets Mar 6, 2024
@consideRatio consideRatio marked this pull request as ready for review March 6, 2024 10:26
@consideRatio consideRatio changed the title Cleanup legacy bucket infrastructure and transition remaining scratch buckets Cleanup config connector historically used to provision gcp buckets Mar 6, 2024
@consideRatio
Copy link
Contributor Author

Thank you @sgibson91 for reviewing!!

I made another commit updating outdated docs, I figure its at least an incremental improvement - so I'll go for a merge. I'll address review feedback if done retroactively!

@consideRatio consideRatio merged commit 85b14f4 into 2i2c-org:master Mar 6, 2024
34 checks passed
@github-actions GitHub Actions
Copy link

github-actions bot commented Mar 6, 2024

🎉🎉🎉🎉

Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/8170616150

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgibson91 sgibson91 sgibson91 approved these changes

Assignees

@consideRatio consideRatio

Labels
None yet
Projects
No open projects
Sprint Board
Status: Done 🎉
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Transition GCP config-connector to terraform - cleanup cloudResources
2 participants
@consideRatio @sgibson91