For reference: work towards adding JMTE hub (AWS, eksctl, cloudformation) #436
Conversation
config/hubs/jmte.cluster.yaml
Outdated
| # we use the node label "k8s.dask.org/node-purpose: worker" | ||
| # specifically for enforce workers to schedule on such nodes. | ||
| # | ||
| traefik: |
There was a problem hiding this comment.
Great! Let's figure a way out to move these to the base!
|
@consideRatio if you'd like, we (or I?) can extract the parts of this PR that aren't related to JMTE and merge them separately. |
|
❤️ THANKS For your energy injection as usual @yuvipanda :) 🎉 Abolutely feel free to go for it! I didnt feel i had the time to prioritize any upstream PRs so i just just tried to make various PRs be commit like chunks. |
|
I've #439 which pulls out commits from here for v1.0-bea bump |
consideRatio
force-pushed
the
pr/add-jmte-and-more
branch
from
203747e to
9c5e6ee
Compare
|
I rebased on master that now includes some of the previous commits. I have no way of creating a kubeconfig that would be functional for more than ~1 day though so this is not something we can start deploying automatically yet. This is related to #381 but a more extreme situation where the ~30 day manual update of kubeconfig isn't possible. |
|
This is awesome work, Erik! Thanks for sharing it! |
|
@consideRatio I extracted another individual commit from this - #455 |
consideRatio
force-pushed
the
pr/add-jmte-and-more
branch
from
a45e1fb to
06566f6
Compare
RebasedI rebased this on current master before doing a redeploy, everything still works. I'm still running in a fork due to AWS permissions etc and to feel freedom to quickly adjust things based on needs. |
|
UPDATE: See #502 about this comment. Extending JMTE to use jupyterhub-sshIf this was a custom deployment I'd manage myself, I'd have a meta helm chart and add another Helm chart to its dependency. I can do this here as well, but it would influence other Helm charts etc. An idea of what can be done is to use a dependency that is disabled by default via a condition set via values. @2i2c-org/tech-team what do you think about me adding a opt-in Helm chart dependency to jupyterhub-ssh in the daskhub helm chart? Btw, it would also make sense to make the basehub helm chart have a opt-in dependency on the dask-gateway helm chart and just have a single meta chart instead of multiple. Reference
|
|
@consideRatio I'm not quite sure what are the implications of your suggestion...it also feels related to #502 right? Should we discuss and converge there? Or perhaps bring this up in a team meeting to brainstorm? |
|
@choldgraf I'm not fully confident about the implications either - how will this influence 2i2c deployment logic etc? I think #502 has a quite clear idea that can be concretely evaluated for its implications. The implications are not meant for the end users, but the 2i2c admins and those wanting to add more charts beyond z2jh and dask-gateway to their deployed 2i2c hub.
Yes to discuss and converge there and yes to bringing this up in a team meeting to brainstorm! :) |
|
For sure - this feels very relevant to ones is the points that we discussed in the Pangeo collaboration meeting as well around allowing binder functionality to be a feature flag of jupyterhub https://discourse.pangeo.io/t/notes-from-the-pangeo-2i2c-kick-off-meeting/1587 As well as that team compass conversation in jupyterhub about multi application deployments |
consideRatio
force-pushed
the
pr/add-jmte-and-more
branch
2 times, most recently
from
bba00ce to
30ea29b
Compare
- Use jsonnet to generate the eksctl config YAML file. This reduces duplication a lot, and matches what we do with kops. We can easily set defaults, generate nodegroup names, and set cloud tags equivalent to our node labels / taints automatically. - Install the cluster autoscaler as part of the support charts, since eksctl doesn't build that by default. It is turned off by default. - No changes were needed for the hub configuration, which is great! Inspired by 2i2c-org#436
- Use jsonnet to generate the eksctl config YAML file. This reduces duplication a lot, and matches what we do with kops. We can easily set defaults, generate nodegroup names, and set cloud tags equivalent to our node labels / taints automatically. - Install the cluster autoscaler as part of the support charts, since eksctl doesn't build that by default. It is turned off by default. - No changes were needed for the hub configuration, which is great! Inspired by 2i2c-org#436
- Use jsonnet to generate the eksctl config YAML file. This reduces duplication a lot, and matches what we do with kops. We can easily set defaults, generate nodegroup names, and set cloud tags equivalent to our node labels / taints automatically. - Install the cluster autoscaler as part of the support charts, since eksctl doesn't build that by default. It is turned off by default. - No changes were needed for the hub configuration, which is great! Inspired by 2i2c-org#436
consideRatio
force-pushed
the
pr/add-jmte-and-more
branch
from
30ea29b to
75563e9
Compare
eksctl [supports](https://eksctl.io/usage/iamserviceaccounts/#usage-with-config-files) creating kubernetes service acocunts bound with [IRSA](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html). We create one with S3 access, and bind it to our notebook and dask pods. This should give them full s3 access. Remove separate eksctl cluster jsonnet object, since it was not doing anything useful. Stolen from 2i2c-org#436 Fixes 2i2c-org#492
|
Ping @choldgraf - description updated with the concrete needs for this PR to get merged! |
consideRatio
force-pushed
the
pr/add-jmte-and-more
branch
from
50dc253 to
8927e31
Compare
|
Merging this PR will trigger the following deployment actions. Support and Staging deployments
Production deployments
|
|
This has been subsumed into our infrastructure now (#2201), except for the SFTP part (#2208). With gratitude to @consideRatio, I close this PR! |
For reference and transparency, this PR represents the state of my work to deploy hub.jupytearth.org using 2i2c configuration etc.
This is what remains to be done for this PR to be mergeable, as updated 2021-08-27.
I've configured the deployment to rely on automatic seeding of some secrets so they don't have to be managed by SOPS etc.
I've installed the
jupyterhub-sshHelm chart, but doing so required me modifying the daskhub Helm chart to install it for me. We need to make charts composable rather than have the current situation that instead have dependencies.For reference: work towards adding JMTE hub (AWS, eksctl, cloudformation) #436 (comment)
Using
eksctl, we can't store a kubeconfig with credentials to access the cluster because they time out in ~1 day. Instead, we must make the deployment script be able to use some service account to acquire access.https://github.com/2i2c-org/pilot-hubs/pull/436/files#diff-27ac079997ab718271acef5b6fef5940182968946a57f78458fef5822e26b9ec