What would be best practice for using Pixi in a shared environment setting?

[mahesh-panchal]

Background

Our group uses a Github project template which includes launch scripts to run various workflows. For the analyses we do, we make a new repository from the template, and then run that on our HPC. The launch scripts all use the same shared conda environment. Pixi, however, implements a project centered approach which would mean we would have the same environment recreated for every new analysis. This is problematic as we have file and file number quotas on our project directories and generally, using conda has been a major culprit in running out of space fast.

Question

So what would be the best practice here in regards to using Pixi? On the one hand each analysis maintains it's own lock for reproducibility, but on the other, for every new project we end up duplicating the same files we already use for other projects and this counts unproductively to our quota. If I understand correctly, the detached-environments doesn't solve this, but just provides an alternate path to store project environments. They wouldn't be shared across analyses from different repositories.

[jjjermiah]

could never find a solution that solves all the issues of working on HPC

the closest idea was to have a shared repository of pixi manifests (initially figured one manifest with any number of environments but that gets difficult to manage)
with detached environments and then invoke the 'pixi shell-hook' across different jobs
this just eventually creates some overhead to manage too
and unless the manifests are versioned + version controlled and recorded for each job, reproducibility isn't as straightforward

[ruben-arts]

This is a valid question, we indeed don't support a workflow that fits this directly.

We have lots of floating ideas on improving this workflow. Currently, the project workflow is heavily focused on reproducibility of a project. But that is hard to combine with a centralized set of dependencies.

There are a few ideas that might help you:

• Creating a distribution: This allows maintainers to give the users a single set of packages that could be used to create projects. This will have as a side effect that the cache also stays small as there are fewer options to download, You can do this by creating a package with a specific set of constraints, if you depend on that package, pixi will limit your users to only download those. Checkout rattler-build to build this distribution package.
• If the activated environment is only for tools like cmake git etc. you could use pixi global for that.
• You could try out py-rattler if you only care about conda packages. This lets you code your environments to give you more control and flexibility. This can also read the pixi.lock file.

We're trying to get a good picture of what the HPC community would need. Some parts we already know of are:

• Layered caching, to accommodate a centrally managed cache and a user based sub cache that contains only the last level of package, as the central cache might not be writable.
• An --offline option to make sure pixi won't reach for the network.
• Distributed caching of tasks and builds
• Shared-cache-aware lock files, (related to this discussion)

If you have more ideas, let us know.

[mahesh-panchal]

Thank you for the ideas. There's some interesting information in there. At the moment, the shared conda environment solution works fine for us. I'm just missing saving the conda-lock file in the analysis repository. I can revisit this in the future. It's just that Pixi is so nice already.

So far the things we're looking for in Pixi are the layered caching, an --offline mode (we also have a secure environment with a local mirror of conda packages), and the shared-cache-aware lock files.