Merge pull request #244 from jnywong/hub-user-image-template

Reinstate hub user image template guide

admin/howto/environment/hub-user-image-template-guide.md

@@ -0,0 +1,141 @@
+(hub-user-image-template-guide/how-to)=
+# Create a custom user image for your hub (advanced)
+
+This advanced guide describes how you can create a custom user image for your community's hub. We recommend [updating a community-maintained upstream image](./update-community-image.md) in the first instance, since this reduces the overall maintenance burden over time. This guide is for advanced users who want complete control over a hub's user image from scratch. It uses [this `hub-user-image-template` repository](https://github.com/2i2c-org/hub-user-image-template) to help you get started.
+
+## 1. Use the template repository
+
+Create a new repository for your user image from the [`hub-user-image-template`](https://github.com/2i2c-org/hub-user-image-template) repository.
+
+1. Go to the [`hub-user-image-template`](https://github.com/2i2c-org/hub-user-image-template) repository.
+2. Click the {guilabel}`Use this template` button located at the top of this repository's GitHub page.
+
+   ```{figure} ../../../images/use-template.png
+   :alt: Use this template
+   ```
+3. This will generate a **copy** of the repository that you can modify as you wish.
+
+## 2. Connect the new repository to [quay.io](https://quay.io/)
+
+The image repository is bundled with a GitHub Action that will build a Docker Image from the configuration you've defined in the repository.
+To use this image, you'll need to connect the repository to an **Image Registry** where we will push images.
+We'll use the [repo2docker-action docs](https://github.com/jupyterhub/repo2docker-action) to guide us.
+
+1. Go to the [repo2docker-action guide's section on pushing to `quay.io`](https://github.com/jupyterhub/repo2docker-action#push-repo2docker-image-to-quayio).
+2. Follow all the instructions (except the last step).
+
+When you have completed these steps, you should have:
+
+- A quay.io repository of the form `quay.io/<quay-username>/<repository-name>`
+- You should have two secrets set on your newly-created GitHub repository:
+  - **`QUAY_USERNAME`**: the user name of the `quay.io` robot account
+  - **`QUAY_PASSWORD`**: the password of the `quay.io` robot account
+
+  ```{figure} ../../../images/secrets.png
+  :alt: Secrets
+  ```
+
+## 3. Enable image pushes to quay.io
+
+Step 2 should have provided the appropriate credentials to push the image to quay.io via GitHub Actions.
+This template repository provides a GitHub Actions workflow that is configured to use these credentials to build and push the image to quay.io, but it needs additional configuration. 
+Below are the steps to configure each option.
+
+### Enable quay.io image push for [build.yaml](https://github.com/2i2c-org/hub-user-image-template/blob/main/.github/workflows/build.yaml)
+
+The [build.yaml](https://github.com/2i2c-org/hub-user-image-template/blob/main/.github/workflows/build.yaml) workflow builds the container image and pushes it to quay.io **if** credentials and image name are properly set.
+This happens on every pushed commit on the main branch of the repo (including when a PR is merged).
+The image is also built for every commit pushed to a Pull Request, but **is not pushed to quay.io by default**.
+
+To enable pushing to the appropriate quay.io repository, edit line 47 of [build.yaml](https://github.com/2i2c-org/hub-user-image-template/blob/main/.github/workflows/build.yaml#L46-L47) and:
+
+1. Uncomment the `IMAGE_NAME` option
+2. Replace `<quay-username>/<repository-name>` with the info of the `quay.io` repository created at step 2
+3. Commit the changes you've made to `build.yaml`
+
+```{figure} ../../../images/image-name-in-build-workflow.png
+:alt: IMAGE_NAME
+```
+
+You can checkout the logs of this GitHub Workflow via the Github Actions tab on your image repository.
+
+```{figure} ../../../images/build-workflow.png
+:alt: Build workflow
+```
+
+If you are triggering this action by merging a PR or directly pushing to the main branch, you should look at the Github Actions tab and this will show a `pushing quay.io/...` message, followed by the image name and tag like in the image below.
+
+```{figure} ../../../images/pushing-to-registry-job-step.png
+:alt: Build logs
+```
+
+### Enable quay.io image **push** during Pull Requests
+
+The [build.yaml](https://github.com/2i2c-org/hub-user-image-template/blob/main/.github/workflows/build.yaml) workflow builds the container image on pull requests.
+It can also push it to quay.io **if** the [`NO_PUSH`](https://github.com/jupyterhub/repo2docker-action#optional-inputs) input is removed.
+
+To enable pushing to the appropriate quay.io repository, edit line 33 of [build.yaml](https://github.com/2i2c-org/hub-user-image-template/blob/main/.github/workflows/build.yaml#L32-L33) and comment out or remove the `NO_PUSH` input.
+
+:::{tip}
+The [Optional Inputs](https://github.com/jupyterhub/repo2docker-action#optional-inputs) section in the [jupyterhub/repo2docker-action](https://github.com/jupyterhub/repo2docker-action) docs provides more details about the `NO_PUSH` option, alongside additional inputs that can also be passed to the repo2docker-action.
+:::
+
+If configured correctly, you will see a `pushing quay.io/...` message in the GitHub Actions logs when you have committed directly to main/merged a PR **and** pushed a commit to a PR.
+
+## 4. Customize the image
+
+Modify [the environment.yml](https://github.com/2i2c-org/hub-user-image-template/blob/main/environment.yml) file and add all the packages you want installed in the conda environment.
+Note that repo2docker already installs [this list](https://github.com/jupyterhub/repo2docker/blob/HEAD/repo2docker/buildpacks/conda/environment.yml) of packages.
+More about what you can do with `environment.yml`, can be found in the [repo2docker docs](https://repo2docker.readthedocs.io/en/latest/config_files.html#environment-yml-install-a-conda-environment).
+
+1. Commit the changes made to `environment.yml`.
+2. Create a Pull Request with this commit, or push it directly to the `main` branch.
+3. If you merge the PR above or directly push the commit to the `main` branch, the GitHub Action will automatically build and push the container image. Wait for this action to finish.
+
+## 5. Build and push the image
+
+Images generated by this action are automatically tagged with both `latest` and `<SHA>` corresponding to the relevant commit SHA on GitHub.
+Both tags are pushed to the image registry specified by the user.
+If an existing image with the *latest* tag already exists in your registry, this Action attempts to pull that image as a cache to reduce unnecessary build steps.
+
+Checkout an example of a [quay.io respository](https://quay.io/repository/2i2c/coessing-image?tab=tags) that hosts the user environment image of a 2i2c hub.
+
+## 6. Connect the hub with this user image
+
+In order to be able to pull the image you need to make sure that your repository is public on your registry. For `quay.io` you can set this under `'Repository Settings'>'Repository Visibility'`
+
+1. Go to the list of image tags on `quay.io`, and find the tag of the last push.
+   This is usually **under the latest tag**.
+   Use this to construct your image name - `quay.io/<quay-username>/<repository-name>:<tag>`.
+
+   ```{figure} ../../../images/coessing-image-quay.png
+   :alt: Tags list example
+   ```
+2. Open the [Configurator](https://pilot.2i2c.org/en/latest/admin/howto/configurator.html) for the hub (you need to be logged in as an admin).
+   You can access it from the hub control panel, under Services in the top bar or by going to `https://<hub-address>/services/configurator/`
+
+   ```{figure} ../../../images/configurator.png
+   :alt: Configurator
+   ```
+3. Make a note of the current image name there.
+4. Put the image tag you constructed in a previous step into the User docker image text box. Note quay web urls look like `quay.io/repository/2i2c/coessing-image`, but the docker image name in configurator **should not** include the repository part of the URL e.g. `quay.io/2i2c/coessing-image:55adca9b2caa`.
+5. Click Submit! *this is alpha level software, so there is no 'has it saved' indicator yet :)*
+
+See [the Configurator docs](https://pilot.2i2c.org/en/latest/admin/howto/configurator.html) for more information.
+
+## 7. Test the new image
+
+Test the new image by starting a new user server!
+If you already had one running, you need to stop and start it again to test.
+If you find new issues, you can revert back to the previous image by entering the old image name in the JupyterHub Configurator.
+
+*This will be streamlined in the future.*
+
+## Appendix: Push the image to a registry other than Quay.io :cloud:
+
+The [jupyterhub/repo2docker-action](https://github.com/jupyterhub/repo2docker-action) can build and push the image to registries other than [Quay.io](https://quay.io/).
+Checkout the [action docs](https://github.com/jupyterhub/repo2docker-action/blob/master/README.md) for the instructions on how to setup your workflow to push to: [AWS Elastic Container Registry](https://github.com/jupyterhub/repo2docker-action#push-repo2docker-image-to-amazon-ecr), [Google Container Registry](https://github.com/jupyterhub/repo2docker-action#push-repo2docker-image-to-google-container-registry) (deprecated but popular), [Google Artifact Registry](https://github.com/jupyterhub/repo2docker-action#push-repo2docker-image-to-google-artifact-registry) (preferred), [Azure Container Registry](https://github.com/jupyterhub/repo2docker-action#push-repo2docker-image-to-azure-container-registry).
+
+:::{note}
+For cloud provider-specific registries, if we are running the cluster on our projects, please contact the 2i2c team to give you credentials for it.
+:::

admin/howto/environment/index.md

@@ -5,61 +5,27 @@ configured environment with base libraries, user interfaces and
 languages installed. This allows them to start working immediately,
 without having to install packages themselves.
 
-
 (environment/custom)=
 ## Customize your user environment
 
-While all hubs [come with a default environment](environment/default), it is possible to create a custom user environment for the hub.
-Here are a few ways that you can do this.
+While all hubs [come with a default environment](environment/default), it is possible to create a custom user environment for the hub. Here are a few ways that you can do this.
 
 (environment:image)=
-### Create your own docker image
-
-Our hubs use [docker images](https://www.docker.com/) to enable a reproducible 
-software environment for all users on the hub.
-You can build and bring your own docker image to the hub, which allows you *full control* over your software environment.
-
-In order to do this, we need to define an environment in a repository, generate a Docker image from that environment, push the Docker image to an image registry, and tell your JupyterHub to pull from that registry.
-See the sections below for more detail.
-
-#### A quick overview
-
-We recommend using the [repo2docker tool](https://repo2docker.readthedocs.io/) to define and build your user environment following best practices for version control.
-This is the tool used by [the Binder project](https://mybinder.org) and is a good standard to follow for defining clear and reproducible computational environments.
+### Create your own image
 
-To use repo2docker to build user environments for your hub, you'll need to:
+Our hubs use [software containers](https://www.docker.com/resources/what-container/) to enable a reproducible 
+software environment for all users on the hub. You can build and bring your own image to the hub, which allows you *full control* over your software environment.
 
-1. **Create a repository** that hosts the files that will define your environment.
-2. **Add files in the repository** that define your user environment.
-   Here are a few good resources for defining these files:
+There are two ways that you can do this:
 
-   - The Binder Project has [documentation about what files are supported](https://mybinder.readthedocs.io/en/latest/using/config_files.html).
-   - The Turing Way has an excellent a collection of tutorials for getting started with Binder, and covers [Python](https://github.com/alan-turing-institute/the-turing-way/blob/main/workshops/boost-research-reproducibility-binder/workshop-presentations/zero-to-binder-python.md), [Julia](https://github.com/alan-turing-institute/the-turing-way/blob/main/workshops/boost-research-reproducibility-binder/workshop-presentations/zero-to-binder-julia.md), and [the R language](https://github.com/alan-turing-institute/the-turing-way/blob/main/workshops/boost-research-reproducibility-binder/workshop-presentations/zero-to-binder-r.md).
-3. **Create a an image registry account**. This will be the place where you store user images once they are built, so that the hub can access them.
-   :::{tip}
-   We recommend using [the quay.io image registry](https://quay.io/) to host your images.
-   This is a public registry service run by [Red Hat](https://www.redhat.com/en) and is reliable to use.
-   :::
-4. **Set up a GitHub Action** to automatically build a docker image using [the repo2docker action](https://github.com/jupyterhub/repo2docker-action), and push it to a registry.
-5. **Configure your hub** to pull the user image from the registry, either via [the configurator](../configurator.md) (available to hubs configured with a single image only) or by [opening a support ticket](../../../support.md) to request that a 2i2c changes the hub configuration for you.
-
-:::{tip}
-By following these steps, you have also created a [Binder-ready repository](https://mybinder.org). We recommend trying to build your reposity using the [*Other...* server profile option](https://docs.2i2c.org/user/howto/specify-unlisted-image/#specify-your-own-image-for-the-software-environment) (if available), or on `mybinder.org` in order to test the environment before updating your default hub configuration.
-:::
-
-#### An in-depth guide
-
-To help you get started, we've created an in-depth how to [Customize a community-maintained upstream image](customize-image) guide and a [template repository](https://github.com/2i2c-org/example-inherit-from-commuity-image) to guide you through the process. You can go to the template repository by clicking the button below.
-
-```{button-link} https://github.com/2i2c-org/hub-user-image-template/blob/main/README.md
-:color: primary
-Go to template repository
-```
+1. [Update a community-maintained upstream image](./update-community-image.md) – add required software packages to an existing community-maintained upstream image. This is the recommended approach for most use cases since this reduces the maintenance burden of keeping the image up to date.
+1. [Build your own custom image from scratch](./hub-user-image-template-guide.md) – build a new image from scratch that contains all the required software packages. This is an advanced approach if you want full control beyond what is available through community-maintained images.
 
 ```{toctree}
 :hidden:
 
-customize-image
+update-community-image
+hub-user-image-template-guide
 ```
 
 ### Temporarily install packages for a session

admin/howto/environment/customize-image.md → admin/howto/environment/update-community-image.md

@@ -1,4 +1,4 @@
-# Customize a community-maintained upstream image
+# Update a community-maintained upstream image (recommended)
 
 This instructional guide shows you how to add packages to a community-maintained upstream image. In this example, we add the [Python package `xarray`](https://docs.xarray.dev/en/stable/) to the [`jupyter/scipy-notebook` image](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) maintained by the [Jupyter Docker Stacks](https://jupyter-docker-stacks.readthedocs.io/en/latest/index.html) community.
 

conf.py

@@ -69,8 +69,6 @@
     "about/service/roles.md": "about/service/shared-responsibility.md",
     "about/service/team.md": "about/service/shared-responsibility.md",
 
-    # Added 2024-06-11
-    "admin/howto/environment/hub-user-image-template-guide.md": "admin/howto/environment/customize-image.md"
 }
 
 # Disable linkcheck for anchors because it throws false errors for any JS anchors