Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UGR GKE as GCP Deployments; minor docs edit #5844

Merged
merged 6 commits into from
Aug 1, 2024

Conversation

gmerritt
Copy link
Contributor

@gmerritt gmerritt commented Jul 7, 2024

Deploy with the .sh scripts. Resource size/number can be changed as needed. Additional node pools may be added by duplicating the node_pool_ugr01.yaml and naming/customizing, and doing likewise with a new .sh script (e.g., copy deploy_node_pool_ugr01.sh) that points to the new node pool's .yaml config.

Deployments are updatable by simply editing config .yaml and re-running corresponding .sh.

Copy link
Contributor

@shaneknapp shaneknapp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

first pass of comments... generally looks great tho!

@shaneknapp
Copy link
Contributor

also, could you run yamllint on your newly created yamls?

@gmerritt
Copy link
Contributor Author

gmerritt commented Jul 8, 2024

also, could you run yamllint on your newly created yamls?

Seems to be running clean! Thanks!

(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml gke_ugresearch.yaml 
(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml node_pool_ugr01.yaml 
(base) MacBook-Pro-2020:configs gregm$

@shaneknapp
Copy link
Contributor

also, could you run yamllint on your newly created yamls?

Seems to be running clean! Thanks!

(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml gke_ugresearch.yaml 
(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml node_pool_ugr01.yaml 
(base) MacBook-Pro-2020:configs gregm$

actually, you should run it this way: yamllint -d relaxed vendor/google/ugr/gke

@gmerritt
Copy link
Contributor Author

gmerritt commented Jul 8, 2024

also, could you run yamllint on your newly created yamls?

Seems to be running clean! Thanks!

(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml gke_ugresearch.yaml 
(base) MacBook-Pro-2020:configs gregm$ yamllint -c ../../../../../.yamllint.yaml node_pool_ugr01.yaml 
(base) MacBook-Pro-2020:configs gregm$

actually, you should run it this way: yamllint -d relaxed vendor/google/ugr/gke

...so we are going looser than the .yamllint.yaml file? Should we remove the .yamllint.yaml from the project? Note this:

$ more ../../../../../.yamllint.yaml
# This config represents running the command `yamllint -d relaxed .` with the
# following extra rules:
#
# new-line-at-end-of-file:
#   level: warning
# trailing-spaces:
#   level: warning
#

Seems you either do -d relaxed or you do -c .yamllint.yaml if you want newline & trailing spaces rules...but yeah it seems that -d, by definition, ignores the .yamllint.yaml file. Is that so?

@shaneknapp
Copy link
Contributor

shaneknapp commented Jul 8, 2024 via email

@gmerritt
Copy link
Contributor Author

gmerritt commented Jul 8, 2024

-d relaxed-but-not-THAT-relaxed

:P

Ok. Well, running clean in both modes!

@shaneknapp
Copy link
Contributor

i'd also recommend putting a README.md in the ugr/ subdir to explain what it's all about. once that's done, let's merge to staging.

@gmerritt
Copy link
Contributor Author

i'd also recommend putting a README.md in the ugr/ subdir to explain what it's all about. once that's done, let's merge to staging.

Ok, added a README.md!

@@ -0,0 +1,17 @@
images:
images:
- name: us-central1-docker.pkg.dev/ucb-datahub-2018/user-images/primary-user-image
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it will serve us better if we have a separate image for this hub. i'd consider cribbing the image config from data100 and using that instead.

@gmerritt
Copy link
Contributor Author

btw -- the ugr cluster is currently undeployed (er, I had it up & down a number of times briefly to test out the scripts, but it is currently gone). I do believe I should deploy it before we move this pr to staging, else datahub may "expect" the named resources to be there, but will not find them!

Oh and -- there are currently tiny VM sizes for core & user pools in the config.

@gmerritt
Copy link
Contributor Author

So yeah I'm not feeling super-great about the custom image...on the one hand, it really is basically a data100 clone, BUT I can't even repo2docker data100 proper (and ditto for multiple other images) without error, either on my Mac or on a GCP linux VM where I've repo2docker'd datahub images successfully in the past...so I don't trust it etc. :/

More time fussing w/ repo2docker than actually making this change, so I'm tossing the custom image edits into this PR as a talking point.

@shaneknapp
Copy link
Contributor

shaneknapp commented Jul 30, 2024 via email

@gmerritt
Copy link
Contributor Author

I'm going merge...tomorrow...so it's more chill.

@gmerritt gmerritt merged commit 1f8596e into berkeley-dsep-infra:staging Aug 1, 2024
23 checks passed
@gmerritt
Copy link
Contributor Author

gmerritt commented Aug 1, 2024

FileNotFoundError: [Errno 2] No such file or directory: 'deployments/ugr01/secrets/gcr-key.json'

Exited with code exit status 1
screenshot_571

@gmerritt
Copy link
Contributor Author

gmerritt commented Aug 1, 2024

I believe that my answer is here, but I'm not sure how to connect the dots...
https://docs.datahub.berkeley.edu/en/latest/admins/credentials.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants