Add support to pass CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY argument for c…

…ontainer_build (#237)
open-ce · Nov 28, 2024 · 9c2fcfb · 9c2fcfb
1 parent 065e00b
commit 9c2fcfb
Show file tree

Hide file tree

Showing 6 changed files with 27 additions and 5 deletions.
diff --git a/README.md b/README.md
@@ -29,7 +29,7 @@ The `open-ce` tool allows a user to build collections of conda recipes described
 * `conda-build` >= 3.22
   * Once `conda` is installed, `conda-build` can be installed with the command: `conda install conda-build`
 * `networkx` >= 2.5
-* `python` >= 3.8
+* `python` >= 3.10
 * `junit-xml` >= 1.9
 * `docker` >= 1.13 or `podman` >= 2.0.5
   * docker or podman required only when building within a container (see below).
@@ -73,7 +73,7 @@ pip install -e .
 | 1.8.1                   | 12.0.1                  |
 | 1.9.4                   | 12.0.3                  |
 | 1.10.0                  | 12.0.3                  |
-| 1.11.4                  | 13.0.2                  |
+| 1.11.4                  | 13.0.3                  |
 
 ### Building a Collection of Packages
 To build an entire integrated and functional conda channel using Open-CE, start by installing the needed tools in the [Requirements](#requirements) section above.
@@ -115,6 +115,14 @@ open-ce build env --python_versions 3.10,3.11 --build_types cuda opence-env
 
 Note that having _conda-forge_ in your channel list may sometime cause conflicts or unexpected errors due to dependencies' versions mismatch. So, it is recommended to avoid mixing the channels during the build as well as at runtime.
 
+### Note:
+Ensure CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0 is set before building setuptools v72.1.0
+The steps to build setuptools with this environment variable and before other packages are:
+1. Set export CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0
+2. Build common-deps.yaml
+3. unset CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY
+4. Build opence-env.yaml.
+Please see [`doc/README.open_ce_build.md`](doc/README.open_ce_build.md#open-ce-build-env-sub-command) if building within a container.
 
 ### Power10 MMA Optimization
 #### Building Packages

diff --git a/doc/README.open_ce_build.md b/doc/README.open_ce_build.md
@@ -95,6 +95,10 @@ The paths to the `env_config_file`s and `--conda_build_configs` must point to
 files within the `open-ce` directory and be relative to the directory
 containing the root level `open-ce` directory.
 
+Note: As setuptools now needs to be built as part of OpenCE. Follow the following steps to build all packages via `--container_build` option:
+1. Pass `--build-arg CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0` to build common-deps.yaml first.
+2. Next pass `--build-arg CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=1` to build opence-env.yaml.
+
 ### Use System MPI
 
 By default, building the entire
@@ -467,7 +471,7 @@ check the Dockerfile for details.
 * GCC Compiler: GCC12
 
 One can build Power10 enabled packages with above system requirements. Note that Power10 is not required on your build system. The libraries can be built on Power9 as well.
-To install GCC 11, following command can be used -
+To install GCC 12, following command can be used -
 ```shell
     yum install -y gcc-toolset-12
 ```
@@ -479,7 +483,7 @@ For example:
     export GCC_HOME=/opt/rh/gcc-toolset-12/root/usr
 ```
 
-GCC 11 setup is automated if the builds are done in a podman container using `--container_build` option. Please see [`Dockerfile`](https://github.com/open-ce/open-ce-builder/blob/main/open_ce/images/builder/Dockerfile-p10) used for containerized build of these packages.
+GCC 12 setup is automated if the builds are done in a podman container using `--container_build` option. Please see [`Dockerfile`](https://github.com/open-ce/open-ce-builder/blob/main/open_ce/images/builder/Dockerfile-p10) used for containerized build of these packages.
 
 #### Build packages
 Power10 MMA Optimization is applicable for cpu only builds. To build Power10 packages, use one of the following options:

diff --git a/open_ce/images/builder-cuda/ubi8/Dockerfile.cuda-11.8 b/open_ce/images/builder-cuda/ubi8/Dockerfile.cuda-11.8
@@ -13,6 +13,9 @@ ARG GROUP_ID=1500
 ENV BUILD_USER=builder
 ARG BUILD_ID=1084
 
+ARG CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0
+ENV CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=${CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY}
+
 ARG LIMIT_BUILD_RESOURCES=0
 ENV LIMIT_BUILD_RESOURCES=${LIMIT_BUILD_RESOURCES}
 

diff --git a/open_ce/images/builder-cuda/ubi8/Dockerfile.cuda-12.2 b/open_ce/images/builder-cuda/ubi8/Dockerfile.cuda-12.2
@@ -13,6 +13,9 @@ ARG GROUP_ID=1500
 ENV BUILD_USER=builder
 ARG BUILD_ID=1084
 
+ARG CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0
+ENV CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=${CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY}
+
 ARG LIMIT_BUILD_RESOURCES=0
 ENV LIMIT_BUILD_RESOURCES=${LIMIT_BUILD_RESOURCES}
 

diff --git a/open_ce/images/builder/ubi8/Dockerfile b/open_ce/images/builder/ubi8/Dockerfile
@@ -12,6 +12,9 @@ ARG GROUP_ID=1500
 ENV BUILD_USER=builder
 ARG BUILD_ID=1084
 
+ARG CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0
+ENV CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=${CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY}
+
 ARG LIMIT_BUILD_RESOURCES=0
 ENV LIMIT_BUILD_RESOURCES=${LIMIT_BUILD_RESOURCES}
 

diff --git a/open_ce/images/builder/ubi8/Dockerfile-p10 b/open_ce/images/builder/ubi8/Dockerfile-p10
@@ -11,7 +11,8 @@ ENV CICD_GROUP=cicd
 ARG GROUP_ID=1500
 ENV BUILD_USER=builder
 ARG BUILD_ID=1084
-
+ARG CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=0
+ENV CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY=${CONDA_ADD_PIP_AS_PYTHON_DEPENDENCY}
 ARG LIMIT_BUILD_RESOURCES=0
 ENV LIMIT_BUILD_RESOURCES=${LIMIT_BUILD_RESOURCES}