Investigate why deployer updates 2i2c staging and not 2i2c $HUB_NAME staging

[jnywong]

Context

When dealing with https://2i2c.freshdesk.com/a/tickets/2479, I noticed that our GH action to deploy from our infrastructure repo that was triggered by #5182 is updating the 2i2c staging hub and not the 2i2c ucmerced staging hub.

See the relevant deployer command in https://github.com/2i2c-org/infrastructure/actions/runs/11976687183/job/33392976328#step:5:58

Task list

Tasks

Beta Give feedback

1. .
2. .
3. .

Definition of Done

No response

[sgibson91]

We actually don't need to do an investigation, I know exactly why this is happening.

We had a design principle that each cluster would only have one staging hub on it, and if you were on a shared cluster, you would share the staging hub. Therefore the deployer only looks for one staging hub per cluster by design. But we've been terrible at enforcing this, as evidenced by the ucmerced-staging hub (which was originally supposed to be temporary I think, but no timeline was put on that so obviously it didn't get removed).

I had a chat with @yuvipanda a while ago about a project where I would refactor all this code and the CI/CD pipeline, decouple the support deployment from the staging deployment, and permit more than one staging hub. And I'd like to do this work before September next year, i.e. RSECon25, where I intend to submit a talk I gave at JupyterCon about our CI/CD system.

[sgibson91]

I will close this out in favour of #5186

[jnywong]

Thanks for the info! I guess it's just "one of those things" that's not written down anywhere because we had the intention to fix it, and it would have been useful to know when dealing with 2i2c $HUB_NAME staging support.

[sgibson91]

Tbf, the principle is documented here and it's noted that 2i2c is a special case because it has a staging and dask-staging https://infrastructure.2i2c.org/reference/ci-cd/hub-deploy/#upgrade-support-and-staging-upgrade-support-and-staging-hub-helm-charts-on-clusters-that-require-it

It's another case of we wrote a policy and didn't enforce it