purge miniconda; add miniforge (#176)

bihealth · Sep 18, 2024 · 335050f · 335050f
1 parent 971f61d
commit 335050f
Show file tree

Hide file tree

Showing 6 changed files with 97 additions and 97 deletions.
diff --git a/bih-cluster/docs/best-practice/software-installation-with-conda.md b/bih-cluster/docs/best-practice/software-installation-with-conda.md
@@ -10,7 +10,10 @@ to do so. We will provide you with a description on how to install conda and how
 to use it. Of course there are many online resources that you can also use.
 Please find a list at the end of the document.
 
-Also note that some system-level software is managed through environment modules.
+!!! Warning
+    Following a change in their terms of service Anaconda Inc. has started to demand payment from research institutions
+    for using both Anaconda, Miniconda, and the defaults channel. As a consequence, 
+    usage of this software is prohibited and we're recommending the alternative free "miniforge" distribution instead.
 
 ## Premise
 When you logged into the cluster, please make sure that you also executed `srun` to log into a computation node and perform the software installation there.
@@ -19,24 +22,21 @@ When you logged into the cluster, please make sure that you also executed `srun`
 
 ```bash
 hpc-login-1:~$ srun --mem=5G --pty bash -i
-hpc-cpu-123:~$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-hpc-cpu-123:~$ bash Miniconda3-latest-Linux-x86_64.sh -b -f -p $HOME/work/miniconda
-hpc-cpu-123:~$ eval "$(/$HOME/work/miniconda/bin/conda shell.bash hook)"
+hpc-cpu-123:~$ wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
+hpc-cpu-123:~$ bash Miniforge3-Linux-x86_64.sh -b -f -p $HOME/work/miniforge
+hpc-cpu-123:~$ eval "$(/$HOME/work/miniforge/bin/conda shell.bash hook)"
 hpc-cpu-123:~$ conda init
 hpc-cpu-123:~$ conda config --set auto_activate_base false
 ```
 
-This will install conda to `$HOME/work/miniconda`.
+This will install conda to `$HOME/work/miniforge`.
 You can change the path to your liking, but please note that your `$HOME` folder has limited space.
 The `work` subfolder however has a bigger quota. More about this [here](../storage/home-quota.md).
 
-To make bioinformatics software available, we have to add the `bioconda` and
-some other channels to the conda configuration:
+To make bioinformatics software available, we have to add the `bioconda` channel to the conda configuration:
 
 ```bash
 hpc-cpu-123:~$ conda config --add channels bioconda
-hpc-cpu-123:~$ conda config --add channels default
-hpc-cpu-123:~$ conda config --add channels conda-forge
 ```
 
 ## Installing software with conda

diff --git a/bih-cluster/docs/how-to/connect/gpu-nodes.md b/bih-cluster/docs/how-to/connect/gpu-nodes.md
@@ -31,14 +31,14 @@ Your job will be automatically placed in the Slurm `gpu` partition and allocated
 
 ## Usage example
 ### Preparation
-We will setup a miniconda installation with `pytorch` testing the GPU.
+We will setup a miniforge installation with `pytorch` testing the GPU.
 If you already have this setup then you can skip this step
 
 ```bash
 hpc-login-1:~$ srun --pty bash
-hpc-cpu-1:~$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-hpc-cpu-1:~$ bash Miniconda3-latest-Linux-x86_64.sh -b -p ~/work/miniconda3
-hpc-cpu-1:~$ source ~/work/miniconda3/bin/activate
+hpc-cpu-1:~$ wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
+hpc-cpu-1:~$ bash Miniforge3-Linux-x86_64.sh -b -p ~/work/miniforge
+hpc-cpu-1:~$ source ~/work/miniforge/bin/activate
 hpc-cpu-1:~$ conda create -y -n gpu-test pytorch cudatoolkit=10.2 -c pytorch
 hpc-cpu-1:~$ conda activate gpu-test
 hpc-cpu-1:~$ python -c 'import torch; print(torch.cuda.is_available())'
@@ -80,7 +80,7 @@ nvcc: NVIDIA (R) Cuda compiler driver
 Copyright (c) 2005-2019 NVIDIA Corporation
 Built on Wed_Oct_23_19:24:38_PDT_2019
 Cuda compilation tools, release 10.2, V10.2.89
-hpc-gpu-1:~$ source ~/work/miniconda3/bin/activate
+hpc-gpu-1:~$ source ~/work/miniforge/bin/activate
 hpc-gpu-1:~$ conda activate gpu-test
 hpc-gpu-1:~$ python -c 'import torch; print(torch.cuda.is_available())'
 True

diff --git a/bih-cluster/docs/how-to/software/tensorflow.md b/bih-cluster/docs/how-to/software/tensorflow.md
@@ -155,7 +155,7 @@ print(tf.nn.softmax(predictions).numpy())
 #SBATCH --partition=gpu
 #SBATCH --gres=gpu:tesla:1
 
-source $HOME/work/miniconda3/bin/activate
+source $HOME/work/miniforge/bin/activate
 conda activate python-tf
 
 python tf_script.py &>tf-out.txt

diff --git a/bih-cluster/docs/ondemand/interactive.md b/bih-cluster/docs/ondemand/interactive.md
@@ -167,7 +167,7 @@ dependencies:
 </details>
 
 
-Starting the Rstudio session via the OnDemand portal works almost as described above (see Example 1). However, you do have to select \`miniconda\` as R source and provide the path to your miniconda installation and (separated by a colon) the name of the (newly created) conda enviroment you want to use.
+Starting the Rstudio session via the OnDemand portal works almost as described above (see Example 1). However, you do have to select \`miniforge\` as R source and provide the path to your miniforge installation and (separated by a colon) the name of the (newly created) conda enviroment you want to use.
 
   ![](figures/ondemand-interactive-custom-conda.png){: style="width:90%;" .center}
 

diff --git a/bih-cluster/docs/overview/storage.md b/bih-cluster/docs/overview/storage.md
@@ -1,79 +1,79 @@
-# Nodes and Storage Volumes
-
-!!! info "No mounting **on** the cluster itself."
-
-    For various technical and security-related reasons it is not possible to mount anything on the cluster nodes by users.
-    For mounting the cluster storage on your computer, please read [Connecting: SSHFS Mounts](../connecting/advanced-ssh/linux.md#file-system-mount-via-sshfs).
-
-This document gives an overview of the nodes and volumes on the cluster.
-
-## Cluster Layout
-
-![](figures/Cluster_Layout.png)
-
-## Cluster Nodes
-
-The following groups of nodes are available to cluster users.
-There are a number of nodes that are invisible to non-admin staff, hosting the queue master and monitoring tools and providing backup storage for key critical data, but these are not shown here.
-
-- `hpc-login-{1,2}`
-    - available as `hpc-login-{1,2}.cubi.bihealth.org`
-    - do not perform any computation on these nodes!
-    - each process may at most use 1 GB of RAM
-- `med0101..0124,0127`
-    - 25 standard nodes
-    - Intel Xeon E5-2650 v2 @2.60Ghz, 16 cores x2 threading
-    - 128 GB RAM
-- `med0133..0164`
-    - 32 standard nodes
-    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
-    - 192 GB RAM
-- `med0201..0264`
-    - 64 nodes with Infiniband interconnect
-    - Intel Xeon E5-2650 v2 @2.60Ghz, 16 cores x2 threading
-    - 128 GB RAM
-- `med0301..0304`
-   - 4 nodes with 4 Tesla V100 GPUs each
-- `med0401..0405` special purpose/high-memory machines
-    - Intel Xeon E5-4650 v2 @2.40GHz, 40 cores x2 threading
-    - `med0401` and `med0402`
-        - 1 TB RAM
-    - `med0403` and `med0404`
-        - 500 GB RAM
-    - `med0405`
-        - 2x "Tesla K20Xm" GPU accelleration cards (cluster resource `gpu`)
-        - access limited to explicit GPU users
-- `med0601..0616`
-    - 16 nodes owned by CUBI
-    - Intel Xeon E5-2640 v3 @2.60Ghz
-    - 192 GB RAM
-- `med0618..0633`
-    - 16 nodes owned by CUBI
-    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
-    - 192 GB RAM
-- `med0701..0764`
-    - 64 standard nodes
-    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
-    - 192 GB RAM
-
-## Cluster Volumes and Locations
-
-The cluster has 2.1 PB of legacy fast storage, currently available at `/fast`, as well as 1.6 PB of next-generation fast storage, available at `/data/cephfs-1`.
-Additionally 7.4 PB of slower "Tier 2" storage is available at `/data/cephfs-2`.
-Storage is provided by a Ceph storage cluster and designed for massively parallel access from an HPC system.
-In contrast to "single server" NFS systems, the system can provide large bandwidth to all cluster nodes in parallel as long as large data means relatively "few" files are read and written.
-
-Storage is split into three sections:
-
-- `home` -- *small, persistent, and safe storage*, e.g., for documents and configuration files (default quota of 1 GB).
-- `work` -- *larger and persistent storage*, e.g., for your large data files (default quota of 1 TB).
-- `scratch` -- *large and non-persistent storage*, e.g., for temporary files, files are automatically deleted after 2 weeks (default quota of 10 TB; deletion not implemented yet).)
-
-Each user, group, and project has one or more of these sections each, e. g. for users:
-
-- `/data/cephfs-1/home/users/$NAME`
-- `/data/cephfs-1/home/users/$NAME/work`
-- `/data/cephfs-1/home/users/$USER/scratch`
-
-
-See [Storage and Volumes: Locations](../storage/storage-locations.md) for more informatin.
+# Nodes and Storage Volumes
+
+!!! info "No mounting **on** the cluster itself."
+
+    For various technical and security-related reasons it is not possible to mount anything on the cluster nodes by users.
+    For mounting the cluster storage on your computer, please read [Connecting: SSHFS Mounts](../connecting/advanced-ssh/linux.md#file-system-mount-via-sshfs).
+
+This document gives an overview of the nodes and volumes on the cluster.
+
+## Cluster Layout
+
+![](figures/Cluster_Layout.png)
+
+## Cluster Nodes
+
+The following groups of nodes are available to cluster users.
+There are a number of nodes that are invisible to non-admin staff, hosting the queue master and monitoring tools and providing backup storage for key critical data, but these are not shown here.
+
+- `hpc-login-{1,2}`
+    - available as `hpc-login-{1,2}.cubi.bihealth.org`
+    - do not perform any computation on these nodes!
+    - each process may at most use 1 GB of RAM
+- `med0101..0124,0127` 
+    - 25 standard nodes
+    - Intel Xeon E5-2650 v2 @2.60Ghz, 16 cores x2 threading
+    - 128 GB RAM
+- `med0133..0164`
+    - 32 standard nodes
+    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
+    - 192 GB RAM
+- `med0201..0264`
+    - 64 nodes with Infiniband interconnect
+    - Intel Xeon E5-2650 v2 @2.60Ghz, 16 cores x2 threading
+    - 128 GB RAM
+- `med0301..0304`
+   - 4 nodes with 4 Tesla V100 GPUs each
+- `med0401..0405` special purpose/high-memory machines
+    - Intel Xeon E5-4650 v2 @2.40GHz, 40 cores x2 threading
+    - `med0401` and `med0402`
+        - 1 TB RAM
+    - `med0403` and `med0404`
+        - 500 GB RAM
+    - `med0405`
+        - 2x "Tesla K20Xm" GPU accelleration cards (cluster resource `gpu`)
+        - access limited to explicit GPU users
+- `med0601..0616`
+    - 16 nodes owned by CUBI
+    - Intel Xeon E5-2640 v3 @2.60Ghz
+    - 192 GB RAM
+- `med0618..0633`
+    - 16 nodes owned by CUBI
+    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
+    - 192 GB RAM
+- `med0701..0764`
+    - 64 standard nodes
+    - Intel Xeon E5-2667 v4 @3.20GHz, 16 cores x 2 threading
+    - 192 GB RAM
+
+## Cluster Volumes and Locations
+
+The cluster has 2.1 PB of legacy fast storage, currently available at `/fast`, as well as 1.6 PB of next-generation fast storage, available at `/data/cephfs-1`.
+Additionally 7.4 PB of slower "Tier 2" storage is available at `/data/cephfs-2`.
+Storage is provided by a Ceph storage cluster and designed for massively parallel access from an HPC system.
+In contrast to "single server" NFS systems, the system can provide large bandwidth to all cluster nodes in parallel as long as large data means relatively "few" files are read and written.
+
+Storage is split into three sections:
+
+- `home` -- *small, persistent, and safe storage*, e.g., for documents and configuration files (default quota of 1 GB).
+- `work` -- *larger and persistent storage*, e.g., for your large data files (default quota of 1 TB).
+- `scratch` -- *large and non-persistent storage*, e.g., for temporary files, files are automatically deleted after 2 weeks (default quota of 10 TB; deletion not implemented yet).)
+
+Each user, group, and project has one or more of these sections each, e. g. for users:
+
+- `/data/cephfs-1/home/users/$NAME`
+- `/data/cephfs-1/home/users/$NAME/work`
+- `/data/cephfs-1/home/users/$USER/scratch`
+
+
+See [Storage and Volumes: Locations](../storage/storage-locations.md) for more informatin.
diff --git a/bih-cluster/docs/slurm/snakemake.md b/bih-cluster/docs/slurm/snakemake.md
@@ -4,8 +4,8 @@ This page describes how to use Snakemake with Slurm.
 
 ## Prerequisites
 
-- This assumes that you have Miniconda properly setup with Bioconda.
-- Also it assumes that you have already activated the Miniconda base environment with `source miniconda/bin/activate`.
+- This assumes that you have Miniforge properly setup with Bioconda.
+- Also it assumes that you have already activated the Miniforge base environment with `source miniforge/bin/activate`.
 
 ## Environment Setup