Multi env proposal documentation (#584)

This proposal came out of a discussion between developers from
prefix.dev, QuantCo and Anaconda.

In this discussion we went over multiple use cases and came to the
conclusion that this approach would give us a good cover of all use
cases we could think of.

This is a design proposal and not a set in stone requirement. We will
iterate on the design while implementing it.

Feel free to give feedback!

This will be visible in the documentation website, this is also the best
way to read it as I used Mkdocs styling to improve readability. (`pixi
run docs` of course 😉 )

Ping: @pavelzw @0xbe7a @travishathaway

---------

Co-authored-by: Pavel Zwerschke <[email protected]>
Co-authored-by: Bela Stoyan <[email protected]>
Co-authored-by: Travis Hathaway <[email protected]>

.github/workflows/rust.yml

@@ -6,10 +6,14 @@ on:
       - main
     paths-ignore:
       - "docs/**"
+      - "mkdocs.yml"
+      - "*.md"
   workflow_dispatch:
   pull_request:
     paths-ignore:
       - "docs/**"
+      - "mkdocs.yml"
+      - "*.md"
 
 name: Rust
 

docs/design_proposals/multi_environment_proposal.md

@@ -0,0 +1,347 @@
+# Proposal Design: Multi Environment Support
+## Objective
+The aim is to introduce an environment set mechanism in the `pixi` package manager.
+This mechanism will enable clear, conflict-free management of dependencies tailored to specific environments, while also maintaining the integrity of fixed lockfiles.
+
+
+### Motivating Example
+There are multiple scenarios where multiple environments are useful.
+
+- **Testing of multiple package versions**, e.g. `py39` and `py310` or polars `0.12` and `0.13`.
+- **Smaller single tool environments**, e.g. `lint` or `docs`.
+- **Large developer environments**, that combine all the smaller environments, e.g. `dev`.
+- **Strict supersets of environments**, e.g. `prod` and `test-prod` where `test-prod` is a strict superset of `prod`.
+- **Multiple machines from one project**, e.g. a `cuda` environment and a `cpu` environment.
+- **And many more.** (Feel free to edit this document in our GitHub and add your use case.)
+
+This prepares `pixi` for use in large projects with multiple use-cases, multiple developers and different CI needs.
+
+## Design Considerations
+1. **User-friendliness**: Pixi is a user focussed tool that goes beyond developers. The feature should have good error reporting and helpful documentation from the start.
+2. **Keep it simple**: Not understanding the multiple environments feature shouldn't limit a user to use pixi. The feature should be "invisible" to the non-multi env use-cases.
+3. **No Automatic Combinatorial**: To ensure the dependency resolution process remains manageable, the solution should avoid a combinatorial explosion of dependency sets. By making the environments user defined and not automatically inferred by testing a matrix of the features.
+4. **Single environment Activation**: The design should allow only one environment to be active at any given time, simplifying the resolution process and preventing conflicts.
+5. **Fixed Lockfiles**: It's crucial to preserve fixed lockfiles for consistency and predictability. Solutions must ensure reliability not just for authors but also for end-users, particularly at the time of lockfile creation.
+
+## Proposed Solution
+!!! important
+    This is a proposal, not a final design. The proposal is open for discussion and will be updated based on the feedback.
+
+### Feature & Environment Set Definitions
+Introduce environment sets into the `pixi.toml` this describes environments based on `feature`'s. Introduce features into the `pixi.toml` that can describe parts of environments.
+As an environment goes beyond just `dependencies` the `features` should be described including the following fields:
+
+- `dependencies`: The conda package dependencies
+- `pypi-dependencies`: The pypi package dependencies
+- `system-requirements`: The system requirements of the environment
+- `activation`: The activation information for the environment
+- `platforms`: The platforms the environment can be run on.
+- `channels`: The channels used to create the environment. Adding the `priority` field to the channels to allow concatenation of channels instead of overwriting.
+- `target`: All the above features but also separated by targets.
+- `tasks`: Feature specific tasks, tasks in one environment are selected as default tasks for the environment.
+
+
+```toml title="Default features" linenums="1"
+[dependencies] # short for [feature.default.dependencies]
+python = "*"
+numpy = "==2.3"
+
+[pypi-dependencies] # short for [feature.default.pypi-dependencies]
+pandas = "*"
+
+[system-requirements] # short for [feature.default.system-requirements]
+libc = "2.33"
+
+[activation] # short for [feature.default.activation]
+scripts = ["activate.sh"]
+```
+
+```toml title="Different dependencies per feature" linenums="1"
+[feature.py39.dependencies]
+python = "~=3.9.0"
+[feature.py310.dependencies]
+python = "~=3.10.0"
+[feature.test.dependencies]
+pytest = "*"
+```
+
+```toml title="Full set of environment modification in one feature" linenums="1"
+[feature.cuda]
+dependencies = {cuda = "x.y.z", cudnn = "12.0"}
+pypi-dependencies = {torch = "1.9.0"}
+platforms = ["linux-64", "osx-arm64"]
+activation = {scripts = ["cuda_activation.sh"]}
+system-requirements = {cuda = "12"}
+# Channels concatenate using a priority instead of overwrite, so the default channels are still used.
+# Using the priority the concatenation is controlled, default is 0, so the default channels are used first.
+channels = [{name = "nvidia", priority = "-1"}, "pytorch"] # Results in:  ["nvidia", "conda-forge", "pytorch"] if the default is `conda-forge`
+tasks = { warmup = "python warmup.py" }
+target.osx-arm64 = {dependencies = {mlx = "x.y.z"}}
+```
+
+
+```toml title="Define tasks as defaults of an environment" linenums="1"
+[feature.test.tasks]
+test = "pytest"
+
+[environments]
+test = ["test"]
+
+# `pixi run test` == `pixi run --environments test test`
+```
+
+The environment definition should contain the following fields:
+
+- `features: Vec<Feature>`: The features that are included in the environment set, which is also the default field in the environments.
+- `solve-group: String`: The solve group is used to group environments together at the solve stage.
+This is useful for environments that need to have the same dependencies but might extend them with additional dependencies.
+For instance when testing a production environment with additional test dependencies.
+
+```toml title="Creating environments from features" linenums="1"
+[environments]
+# implicit: default = ["default"]
+default = ["py39"] # implicit: default = ["py39", "default"]
+py310 = ["py310"] # implicit: py310 = ["py310", "default"]
+test = ["test"] # implicit: test = ["test", "default"]
+test39 = ["test", "py39"] # implicit: test39 = ["test", "py39", "default"]
+```
+
+```toml title="Testing a production environment with additional dependencies" linenums="1"
+[environments]
+# Creating a `prod` environment which is the minimal set of dependencies used for production.
+prod = {features = ["py39"], solve-group = "prod"}
+# Creating a `test_prod` environment which is the `prod` environment plus the `test` feature.
+test_prod = {features = ["py39", "test"], solve-group = "prod"}
+# Using the `solve-group` to solve the `prod` and `test_prod` environments together
+# Which makes sure the tested environment has the same version of the dependencies as the production environment.
+```
+
+```toml title="Creating environments without a default environment" linenums="1"
+[dependencies]
+# Keep empty or undefined to create an empty environment.
+
+[feature.base.dependencies]
+python = "*"
+
+[feature.lint.dependencies]
+pre-commit = "*"
+
+[environments]
+# Create a custom default
+default = ["base"]
+# Create a custom environment which only has the `lint` feature as the default feature is empty.
+lint = ["lint"]
+```
+
+### Lockfile Structure
+Within the `pixi.lock` file, a package may now include an additional `environments` field, specifying the environment to which it belongs.
+To avoid duplication the packages `environments` field may contain multiple environments so the lockfile is of minimal size.
+```yaml
+- platform: linux-64
+  name: pre-commit
+  version: 3.3.3
+  category: main
+  environments:
+    - dev
+    - test
+    - lint
+  ...:
+- platform: linux-64
+  name: python
+  version: 3.9.3
+  category: main
+  environments:
+    - dev
+    - test
+    - lint
+    - py39
+    - default
+  ...:
+```
+
+
+### User Interface Environment Activation
+Users can manually activate the desired environment via command line or configuration.
+This approach guarantees a conflict-free environment by allowing only one feature set to be active at a time.
+For the user the cli would look like this:
+
+```shell title="Default behavior"
+pixi run python
+# Runs python in the `default` environment
+```
+
+```shell title="Activating an specific environment"
+pixi run -e test pytest
+pixi run --environment test pytest
+# Runs `pytest` in the `test` environment
+```
+
+```shell title="Activating a shell in an environment"
+pixi shell -e cuda
+pixi shell --environment cuda
+# Starts a shell in the `cuda` environment
+```
+```shell title="Running any command in an environment"
+pixi run -e test any_command
+# Runs any_command in the `test` environment which doesn't require to be predefined as a task.
+```
+
+```shell title="Interactive selection of environments if task is in multiple environments"
+# In the scenario where test is a task in multiple environments, interactive selection should be used.
+pixi run test
+# Which env?
+# 1. test
+# 2. test39
+```
+
+## Important links
+- Initial writeup of the proposal: https://gist.github.com/0xbe7a/bbf8a323409be466fe1ad77aa6dd5428
+- GitHub project: https://github.com/orgs/prefix-dev/projects/10
+
+## Real world example use cases
+??? tip "Polarify test setup"
+    In `polarify` they want to test multiple versions combined with multiple versions of polars.
+    This is currently done by using a matrix in GitHub actions.
+    This can be replaced by using multiple environments.
+
+    ```toml title="pixi.toml"
+    [project]
+    name = "polarify"
+    # ...
+    channels = ["conda-forge"]
+    platforms = ["linux-64", "osx-arm64", "osx-64", "win-64"]
+
+    [tasks]
+    postinstall = "pip install --no-build-isolation --no-deps --disable-pip-version-check -e ."
+
+    [dependencies]
+    python = ">=3.9"
+    pip = "*"
+    polars = ">=0.14.24,<0.21"
+
+    [feature.py39.dependencies]
+    python = "3.9.*"
+    [feature.py310.dependencies]
+    python = "3.10.*"
+    [feature.py311.dependencies]
+    python = "3.11.*"
+    [feature.py312.dependencies]
+    python = "3.12.*"
+    [feature.pl017.dependencies]
+    polars = "0.17.*"
+    [feature.pl018.dependencies]
+    polars = "0.18.*"
+    [feature.pl019.dependencies]
+    polars = "0.19.*"
+    [feature.pl020.dependencies]
+    polars = "0.20.*"
+
+    [feature.test.dependencies]
+    pytest = "*"
+    pytest-md = "*"
+    pytest-emoji = "*"
+    hypothesis = "*"
+    [feature.test.tasks]
+    test = "pytest"
+
+    [feature.lint.dependencies]
+    pre-commit = "*"
+    [feature.lint.tasks]
+    lint = "pre-commit run --all"
+
+    [environments]
+    pl017 = ["pl017", "py39", "test"]
+    pl018 = ["pl018", "py39", "test"]
+    pl019 = ["pl019", "py39", "test"]
+    pl020 = ["pl020", "py39", "test"]
+    py39 = ["py39", "test"]
+    py310 = ["py310", "test"]
+    py311 = ["py311", "test"]
+    py312 = ["py312", "test"]
+    ```
+
+    ```yaml title=".github/workflows/test.yml"
+    jobs:
+      tests:
+      name: Test ${{ matrix.environment }}
+      runs-on: ubuntu-latest
+      strategy:
+        matrix:
+          environment:
+            - pl017
+            - pl018
+            - pl019
+            - pl020
+            - py39
+            - py310
+            - py311
+            - py312
+      steps:
+        - uses: actions/checkout@v4
+        - uses: prefix-dev/[email protected]
+          with:
+            # already installs the corresponding environment and caches it
+            environments: ${{ matrix.environment }}
+        - name: Install dependencies
+          run: |
+            pixi run --env ${{ matrix.environment }} postinstall
+            pixi run --env ${{ matrix.environment }} test
+    ```
+??? tip "Test vs Production example"
+    This is an example of a project that has a `test` feature and `prod` environment.
+    The `prod` environment is a production environment that contains the run dependencies.
+    The `test` feature is a set of dependencies and tasks that we want to put on top of the previously solved `prod` environment.
+    This is a common use case where we want to test the production environment with additional dependencies.
+
+    ```toml title="pixi.toml"
+    [project]
+    name = "my-app"
+    # ...
+    channels = ["conda-forge"]
+    platforms = ["osx-arm64", "linux-64"]
+
+    [tasks]
+    postinstall-e = "pip install --no-build-isolation --no-deps --disable-pip-version-check -e ."
+    postinstall = "pip install --no-build-isolation --no-deps --disable-pip-version-check ."
+    dev = "uvicorn my_app.app:main --reload"
+    serve = "uvicorn my_app.app:main"
+
+    [dependencies]
+    python = ">=3.12"
+    pip = "*"
+    pydantic = ">=2"
+    fastapi = ">=0.105.0"
+    sqlalchemy = ">=2,<3"
+    uvicorn = "*"
+    aiofiles = "*"
+
+    [feature.test.dependencies]
+    pytest = "*"
+    pytest-md = "*"
+    pytest-asyncio = "*"
+    [feature.test.tasks]
+    test = "pytest --md=report.md"
+
+    [environments]
+    # both default and prod will have exactly the same dependency versions when they share a dependency
+    default = {features = ["test"], solve-group = "prod-group"}
+    prod = {features = [], solve-group = "prod-group"}
+    ```
+    In ci you would run the following commands:
+    ```shell
+    pixi run postinstall-e && pixi run test
+    ```
+    Locally you would run the following command:
+    ```shell
+    pixi run postinstall-e && pixi run dev
+    ```
+
+    Then in a Dockerfile you would run the following command:
+    ```dockerfile title="Dockerfile"
+    FROM ghcr.io/prefix-dev/pixi:latest # this doesn't exist yet
+    WORKDIR /app
+    COPY . .
+    RUN pixi run --env prod postinstall
+    EXPOSE 8080
+    CMD ["/usr/local/bin/pixi", "run", "--env", "prod", "serve"]
+    ```

mkdocs.yml

@@ -81,6 +81,8 @@ nav:
       - C++/Cmake: examples/cpp-sdl.md
       - OpenCV: examples/opencv.md
       - ROS2: examples/ros2-nav2.md
+  - Design Proposals:
+      - Multi Env: design_proposals/multi_environment_proposal.md
   - Community: Community.md
   - FAQ: FAQ.md