Kubeflow-Containers explore multi-stage builds

[EveningStarlight]

Explore the use of multi-stage builds in Docker.
See if they can be used to:

• help organize architecture
• improve build speed
• improve clarity

[EveningStarlight]

The current dockerfile usage for each image

Colouring is just for noticing related blocks.

[EveningStarlight]

An alternative build order, with possible build stages

This could simplify all the jupyterlab and rstudio builds. Letting them share stages, cached resources, and improve build speeds.

[EveningStarlight]

Structure would allow for all images to be stored in one Dockerfile.

Only using the FROM argument simplifies the Jupyter builds for the early stages.
Each stage just needs to reference the build chain and label the current stage.
ex.
FROM base-cpu as base-jupyter
This is a stage name base-jupyter with the base image base-cpu

The limitation of this is that the finalize stage will need to be duplicated for each final image.

[EveningStarlight]

The COPY instruction can be used to copy files and directories from one stage to another. This could be used to start sharing some steps with sas and remote-desktop and reduce the duplication of code.

[EveningStarlight]

Arguments used in FROM statements must be declared at the top of the dockerfile.
The specific case is

ARG BASE_VERSION=2024-06-17
FROM quay.io/jupyter/datascience-notebook:$BASE_VERSION

This line is used for the cpu base and for sas. If they will always use the same base, this will work fine for now.
But it can't be defined separately, so if they need to be different, they need to be renamed.
Then they also need to be declared at the top of the file, before all the FROM statements start getting used. If not, the variable will read as blank and the build will fail.
See
https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/12032558256/job/33544669100

  1684 |     ARG BASE_VERSION_SAS=2024-06-17
 1685 | >>> FROM quay.io/jupyter/datascience-notebook:$BASE_VERSION_SAS as sas
 1686 |     
 1687 |     USER root
--------------------
ERROR: failed to solve: failed to parse stage name "quay.io/jupyter/datascience-notebook:": invalid reference format

The vaiable can't be declared at the top and redefined later either
See
https://github.com/StatCan/aaw-kubeflow-containers/actions/runs/12032666248/job/33545016945

  7     |     ARG BASE_VERSION_SAS=2024-06-17-test
.
.
.
 1686 |     ARG BASE_VERSION_SAS=2024-06-17
 1687 | >>> FROM quay.io/jupyter/datascience-notebook:$BASE_VERSION_SAS as sas
 1688 |     
 1689 |     USER root
--------------------
ERROR: failed to solve: quay.io/jupyter/datascience-notebook:2024-06-17-test: failed to resolve source metadata for quay.io/jupyter/datascience-notebook:2024-06-17-test: quay.io/jupyter/datascience-notebook:2024-06-17-test: not found

[EveningStarlight]

DOCKER_BUILDKIT=0 will break simple builds for multi-stage.
While the legacy build is enabled, the build will build every stage up till the target stage. This greatly balloons the build process