-
Notifications
You must be signed in to change notification settings - Fork 38
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #6511 from berkeley-dsep-infra/staging
merging 6503/6505/6508 to prod
- Loading branch information
Showing
5 changed files
with
175 additions
and
48 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -25,7 +25,7 @@ jupyterhub: | |
| hub.jupyter.org/pool-name: core-pool-2024-05-08 | ||
| config: | ||
| JupyterHub: | ||
| admin_access: false | ||
| admin_access: true | ||
| authenticator_class: dummy | ||
| Authenticator: | ||
| admin_users: | ||
|
|
@@ -35,6 +35,7 @@ jupyterhub: | |
| - felder | ||
| - balajialwar | ||
| - gmerritt | ||
| - [email protected] | ||
| proxy: | ||
| chp: | ||
| nodeSelector: | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,68 @@ | ||
| --- | ||
| title: Test User Images Locally | ||
| --- | ||
|
|
||
| You should use `repo2docker` to build and test the image on your own device before you push and create a PR. It is often faster to do this first before using CI/CD since you can take advantage of local caching and rapid iteration. There's no need to waste Github Action minutes to test build images when you can do this on your own device. | ||
|
|
||
| ## Common Usage | ||
|
|
||
| One can simply run `repo2docker /path/to/image/assets`. For example if one has changed into the directory containing the `repo2docker` files (such as `environment.yml` and/or `Dockerfile`), the command would be: | ||
|
|
||
| ```shell | ||
| repo2docker . | ||
| ``` | ||
|
|
||
| This works on Linux and Windows Subsystem for Linux (WSL). It will build the image, then launch jupyter server and display a localhost URL. Copy the URL and paste it into a local web browser. | ||
|
|
||
| If you just want to build the image without also running the server, | ||
| add the `--no-run` argument: | ||
|
|
||
| ```shell | ||
| repo2docker --no-run . | ||
| ``` | ||
|
|
||
| ## On Apple Silicon | ||
|
|
||
| Apple's ARM-based CPUs (the "M" chips) are different from those run on the virtual machines in our clusters. macOS is capable of emulating x86_64/amd64, but it is necessary to optimize docker for this emulation, and to explicitly tell your local docker runtime that the images should be built on the `linux/amd64` platform. | ||
|
|
||
| In Docker's settings: | ||
|
|
||
| - Under **General** > **Virtual Machine Options**, either enable both **Apple Virtualization framework** and **Use Rosetta for x86_64/amd64 emulation on Apple Silicon**, or enable **Docker VMM**. | ||
| - Under **Resources** it is also recommended to raise the memory limit to at least 4GB. | ||
|
|
||
| There are two methods for building `linux/amd64` images. The default uses `repo2docker`'s support for `docker-py`, while the second uses a `repo2docker` plugin that can invoke your local docker command-line interface. | ||
|
|
||
| ### docker-py (default) | ||
|
|
||
| Run `jupyter-repo2docker` with the following arguments: | ||
|
|
||
| ``` | ||
| repo2docker \ | ||
| --Repo2Docker.platform=linux/amd64 \ | ||
| -e PLAYWRIGHT_BROWSERS_PATH=/srv/conda \ | ||
| --user-id=1000 --user-name=jovyan \ | ||
| --target-repo-dir=/home/jovyan/.cache \ | ||
| . | ||
| ``` | ||
|
|
||
| where the final parameter is the path to the assets or `.` if they are in the current directory. | ||
|
|
||
| The `--user-id` and `--user-name` options are for non-Dockerfile based builds. Images with Dockerfiles do not need those options. | ||
|
|
||
| Note that you may see (possibly harmless) architecture mismatch warnings with this method. | ||
|
|
||
| ### `docker` CLI | ||
|
|
||
| You can instruct `repo2docker` to use your machine's local `docker` executable directly rather than the default of `docker-py`. You will first need to install [repo2podman](https://github.com/manics/repo2podman), a plugin that lets you use any container runtime with a command-line user interface similar to that of `docker`. This is useful if you want to leverage [docker buildx](https://github.com/docker/buildx/) (for things like multi-stage builds) or if you want to use an alternative executable like `podman`. This also eliminates architecture mismatch warnings. | ||
|
|
||
| ::: {.callout-warning} | ||
| repo2podman reportedly does not work yet on WSL. | ||
| ::: | ||
|
|
||
| ``` | ||
| repo2docker \ | ||
| --Repo2Docker.platform=linux/amd64 \ | ||
| -e PLAYWRIGHT_BROWSERS_PATH=/srv/conda \ | ||
| --engine podman --PodmanEngine.podman_executable=docker \ | ||
| . | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,57 @@ | ||
| --- | ||
| title: "DataHub Semester Start and End Tasks" | ||
| --- | ||
|
|
||
| This document outlines the tasks for preparing DataHub for the start of a semester and for concluding semester activities. | ||
|
|
||
| ## Semester Start Tasks | ||
|
|
||
| ### 1. Setup and Configuration | ||
| - [ ] **Bump Replica Values**: Bump replica values in values.yaml | ||
| - [ ] **Set Node Count**: Set the appropriate number of node count for each node pool in GKE console | ||
|
|
||
| ### 2. User Management | ||
| - [ ] **Identify Unused Hubs**: Identify hubs that will not be used during a particular semester | ||
| - [ ] **Send Onboarding Email**: Send onboarding instructions to all the instructors and GSIs added to `datahub-announce` listserv | ||
|
|
||
| --- | ||
|
|
||
| ## Semester End Tasks | ||
|
|
||
| ### 1. Operational Tasks | ||
| - [ ] **Update Kubernetes**: Check for and apply updates to Kubernetes | ||
| - [ ] **Update Ubuntu Single User Images**: | ||
| - [ ] Ubuntu and rocker base image. | ||
| - [ ] Python | ||
| - [ ] R/RStudio (for non-rocker based images) | ||
| - [ ] JupyterHub: Check for and apply updates to Z2JH JupyterHub (if required). | ||
| - [ ] JupyterLab/Notebook: Check for and apply updates to JupyterLab and Notebook (if required) | ||
| - [ ] Conda-forge distribution | ||
| - [ ] Otter Grader: Check for and apply updates to Otter Grader across all hubs (if required) | ||
| - [ ] Quarto | ||
| - [ ] **Update Hub Image**: | ||
| - [ ] JupyterHub: Check for and apply updates to Z2JH JupyterHub (if required). | ||
| - [ ] oauthenticator | ||
| - [ ] ltiauthenticator | ||
| - [ ] **Reduce Resources**: | ||
| - [ ] Scale down node placeholder pods to 0 | ||
| - [ ] Reduce the number of nodes allocated for each node pool | ||
| - [ ] **Clear SQLite DB**: Clear SQLite database that caches user info for hub pods | ||
| - [ ] **Archive User Data**: Archive user home directories across hubs (if required) | ||
| - [ ] **Resize/Consolidate Filestores**: Resize/Consolidate filestore based on the storage snapshot (if required) | ||
| - [ ] **Remove Config**: Remove stanzas added to provide elevated privileges to instructors, increased RAM for courses, shared directories etc.. | ||
| - [ ] **Remove Packages**: Remove packages that were requested for the previous term or older. | ||
| - [ ] **Remove Calendar Events**: Remove calendar events added to support courses in `DataHub Scaling Events` | ||
| - [ ] **Resolve Alerts**: Resolve any dependabot alerts reported | ||
| - [ ] **Version Packages**: Version any packages that are unversioned in environment.yml file | ||
| - [ ] **Update Postgres**: Check for and apply updates to Postgres server and client (if required) | ||
| - [ ] **Create Tokens**: Create a new github [personal access token](https://docs.datahub.berkeley.edu/admins/cicd-github-actions.html#organization-secrets) for our CI/CD pipeline | ||
|
|
||
| ### 2. User Communication | ||
| - [ ] **Backup Data**: Notify users to back up their own files. | ||
| - [ ] **Maintenance Window**: Decide and communicate Maintenance Window (MW) dates with users | ||
|
|
||
| ### 3. Review | ||
| - [ ] **Audit Hubs**: Audit courses and identify the ones that doesn't need their own hub | ||
| - [ ] **Gather feedback**: If necessary, gather feedback about any features piloted during the semester | ||
| - [ ] **Update documentation**: Review documentation and keep it up to date |