From d655f5743ba2f862e6e0dd939ae2e20f21453d13 Mon Sep 17 00:00:00 2001 From: Douglas Jacobsen Date: Fri, 6 Sep 2024 09:13:20 -0600 Subject: [PATCH 1/3] Split requirements into a requirements-dev This commit adds a new requirements-dev.txt file for capturing requirements that are only needed if you want to develop Ramble. --- requirements-dev.txt | 6 ++++++ requirements.txt | 5 ----- 2 files changed, 6 insertions(+), 5 deletions(-) create mode 100644 requirements-dev.txt diff --git a/requirements-dev.txt b/requirements-dev.txt new file mode 100644 index 000000000..bb1e1f3c6 --- /dev/null +++ b/requirements-dev.txt @@ -0,0 +1,6 @@ +-r requirements.txt +pytest +flake8 +black +coverage +pre-commit diff --git a/requirements.txt b/requirements.txt index d5a0cdc05..b860dcfd9 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,10 +1,5 @@ -pytest -flake8 -black google-cloud-storage # for gcs fetch test google-api-core # for gcs fetch error .execptions -coverage -pre-commit graphlib-backport;python_version<"3.9" urllib3==1.26.18;python_version<="3.6" protobuf;python_version>"3.6" From 78358af3977ce5d3b27638668641bbbdc374b660 Mon Sep 17 00:00:00 2001 From: Douglas Jacobsen Date: Fri, 6 Sep 2024 09:25:06 -0600 Subject: [PATCH 2/3] Update dependency documentation --- README.md | 19 ++++++++++++++++--- lib/ramble/docs/package_managers.rst | 21 +++++++++++++++++++++ 2 files changed, 37 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 2c1518dcf..9cb2a49d5 100644 --- a/README.md +++ b/README.md @@ -34,9 +34,14 @@ support Python 3.6 as it is used by older operating systems such as Centos7. Specifically, you might need to update `pip` and downgrade `protobuf` when using Python 3.6. -Outside of these requirements, ramble requires an existing installation of -spack for some application definition. See -[Spack’s documentation](https://github.com/spack/spack#-spack) to install Spack. +Outside of these requirements, Ramble requires package managers to be +configured if they will be used as part of the experiments Ramble creates. + +Although package manager support is not required to use Ramble, some +experiments are more easily accomplished by allowing Ramble to drive the +package manager execution. To allow this, package managers generally need to be +installed indepednently from Ramble. For more information on this, see +[Ramble's supported package managers](https://ramble.readthedocs.io/en/latest/package_managers.html). Documentation ---------------- @@ -65,6 +70,14 @@ Resources: Contributing ------------------------ +When developing features for Ramble, it can be helpful to install the +development requirements instead of the user requirements: + +e.g. + + pip install -r requirements-dev.txt + + Contributing to Ramble is relatively easy. Just send us a [pull request](https://help.github.com/articles/using-pull-requests/). When you send your request, make ``develop`` the destination branch on the diff --git a/lib/ramble/docs/package_managers.rst b/lib/ramble/docs/package_managers.rst index e75f0662b..25b5deb6c 100644 --- a/lib/ramble/docs/package_managers.rst +++ b/lib/ramble/docs/package_managers.rst @@ -75,6 +75,11 @@ into each experiment, to prepare the execution environment. The :ref:`software configuration section ` is used to determine what modules to load within the resulting environment. +The use of this package manager requires an installation of +``environment-modules`` outside of Ramble. This is handled by system +administrators on several clusters, but for more information see +`environment-modules's documentation `. + ^^^^^^^^^^^^^^^^^^^^^ EESSI Package Manager ^^^^^^^^^^^^^^^^^^^^^ @@ -87,6 +92,10 @@ EESSI, and load the correct module files for the execution environment. The :ref:`software configuration section ` is used to determine what modules to load within the resulting environment. +The use of this package manager requires an installation of EESSI outside of +Ramble. For more information, see +`EESSI's documentation `. + ^^^^^^^^^^^^^^^^^^^ Pip Package Manager ^^^^^^^^^^^^^^^^^^^ @@ -101,6 +110,11 @@ manager. The :ref:`software configuration section ` is used to determine what packages to install within the resulting virtual environment. +The use of this package manager requires ``pip`` to be installed outside of +Ramble. This happens automatically in several Python installations. For more +information see +`pip's documentation`. + ^^^^^^^^^^^^^^^^^^^^^ Spack Package Manager ^^^^^^^^^^^^^^^^^^^^^ @@ -118,6 +132,10 @@ When using the Spack package manager, workspaces can also use the ``push-to-cache`` and ``mirror`` pipelines to cache compiled binaies, and mirror software source. +The use of this package manager requires an external installation of Spack. For +instructions on installing Spack, see +`Spack's documentation `. + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Spack Lightweight Package Manager @@ -137,3 +155,6 @@ When using the Spack package manager, workspaces can also use the ``push-to-cache`` and ``mirror`` pipelines to cache compiled binaies, and mirror software source. +The use of this package manager requires an external installation of Spack. For +instructions on installing Spack, see +`Spack's documentation `. From 6a9dccbb640faa049116e5d7a5a6a0ebdfb9d440 Mon Sep 17 00:00:00 2001 From: Douglas Jacobsen Date: Fri, 6 Sep 2024 09:28:07 -0600 Subject: [PATCH 3/3] Update Dockerfiles to capture new requirements --- share/ramble/cloud-build/Dockerfile-apt | 3 ++- share/ramble/cloud-build/Dockerfile-yum | 3 ++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/share/ramble/cloud-build/Dockerfile-apt b/share/ramble/cloud-build/Dockerfile-apt index 82de4b95f..d24bd7207 100644 --- a/share/ramble/cloud-build/Dockerfile-apt +++ b/share/ramble/cloud-build/Dockerfile-apt @@ -17,7 +17,8 @@ RUN cd /opt && \ . spack/share/spack/setup-env.sh && \ spack load py-pip ^python@${PYTHON_VER} && \ wget https://raw.githubusercontent.com/GoogleCloudPlatform/ramble/develop/requirements.txt && \ - python -m pip install -r /opt/requirements.txt + wget https://raw.githubusercontent.com/GoogleCloudPlatform/ramble/develop/requirements-dev.txt && \ + python -m pip install -r /opt/requirements-dev.txt FROM ${BASE_IMG}:${BASE_VER} diff --git a/share/ramble/cloud-build/Dockerfile-yum b/share/ramble/cloud-build/Dockerfile-yum index 8399de069..c1c141329 100644 --- a/share/ramble/cloud-build/Dockerfile-yum +++ b/share/ramble/cloud-build/Dockerfile-yum @@ -17,7 +17,8 @@ RUN cd /opt && \ . spack/share/spack/setup-env.sh && \ spack load py-pip ^python@${PYTHON_VER} && \ wget https://raw.githubusercontent.com/GoogleCloudPlatform/ramble/develop/requirements.txt && \ - python -m pip install -r /opt/requirements.txt + wget https://raw.githubusercontent.com/GoogleCloudPlatform/ramble/develop/requirements-dev.txt && \ + python -m pip install -r /opt/requirements-dev.txt FROM ${BASE_IMG}:${BASE_VER}