more cleanup and re-arrangements

bih-cluster/docs/best-practice/project-structure.md

@@ -1,6 +1,7 @@
 # Project File System Structure
 
-This Wiki page dscribes best pratices for managing your Bioinformatics projects on the file system.
+!!! warning "Under Construction"
+    This guide was written for the old GPFS file system is in the process of being updated.
 
 ## General Aims
 

bih-cluster/docs/best-practice/software-installation-with-conda.md

@@ -1,92 +1,76 @@
 # Software Installation with Conda
-
 ## Conda
-
-For the management of the bioinformatics software on the BIH cluster we are using conda.
-Conda is a package management system that is based on channels, and one of those
-channels provides a huge selection of bioinformatics software.
-
-Conda is written in Python and is based on recipes, such that everybody can
-write recipes for missing software (if there is any). In general the packages
-are pre-compiled and conda just downloads the binaries from the conda servers.
+Users do not have the rights to install system packages on the BIH HPC cluster.
+For the management of bioinformatics software we therefore recommend using the conda package manager.
+Conda provides software in different “channels” and one of those channels contains a huge selection of bioinformatics software (bioconda).
+Generally packages are pre-compiled and conda just downloads the binaries from the conda servers.
 
 You are in charge of managing your own software stack, but conda makes it easy
 to do so. We will provide you with a description on how to install conda and how
 to use it. Of course there are many online resources that you can also use.
 Please find a list at the end of the document.
 
 Also note that some system-level software is managed through environment modules.
-See [System-near Software Provided by HPC Administration](#system-near-software-provided-by-hpc-administration) below.
 
 ## Premise
-
 When you logged into the cluster, please make sure that you also executed `srun` to log into a computation node and perform the software installation there.
 
 ## Installing conda
 
 ```bash
 hpc-login-1:~$ srun --mem=5G --pty bash -i
-med0127:~$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-med0127:~$ bash Miniconda3-latest-Linux-x86_64.sh -b -f -p $HOME/work/miniconda
+hpc-cpu-123:~$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+hpc-cpu-123:~$ bash Miniconda3-latest-Linux-x86_64.sh -b -f -p $HOME/work/miniconda
+hpc-cpu-123:~$ conda init
 ```
 
 This will install conda to `$HOME/work/miniconda`.
 This path can be changed to your liking.
-Please note that the `$HOME` folder has limited space (an exception is the subfolder `$HOME/work` which has no space limit).
-
-NB: `$HOME/scratch` is not appropriate as files placed there will be removed automatically after 2 weeks.
-
-To make it available upon login, extend and export the `$PATH` variable with the
-installation path + `/bin` and add it to your `$HOME/.bashrc`:
-
-```bash
-case "${SLURMD_NODENAME-${HOSTNAME}}" in
-    login-*)
-        ;;
-    *)
-        export PATH=$HOME/work/miniconda/condabin:$PATH
-        ;;
-esac
-```
+Please note that the `$HOME` folder has limited space, [more about that here](../storage/home-quota.md).
 
-The above code makes sure that you don't have conda available on the login nodes,
-where you are not allowed to start any computations.
+!!! note
+    `$HOME/scratch` is not appropriate as files placed there will be removed automatically after 2 weeks.
 
 To make bioinformatics software available, we have to add the `bioconda` and
 some other channels to the conda configuration:
 
 ```bash
-med0127:~$ conda config --add channels bioconda
-med0127:~$ conda config --add channels default
-med0127:~$ conda config --add channels conda-forge
+hpc-cpu-123:~$ conda config --add channels bioconda
+hpc-cpu-123:~$ conda config --add channels default
+hpc-cpu-123:~$ conda config --add channels conda-forge
 ```
 
-You can also add channels to your liking.
+!!! warning "Important"
+    By default conda will automatically activate the (base) environment for new shell sessions.
+    This is can significantly delay your login process and should be turned off:
 
-## Installing software with conda
+    ```sh
+    hpc-cpu-123:~$ conda config --set auto_activate_base false
+    ```
 
+
+## Installing software with conda
 Installing packages with conda is straight forward:
 
 ```bash
-med0127:~$ conda install <package>
+hpc-cpu-123:~$ conda install <package>
 ```
 
-This will install a package into the conda root environment. We will explain
-environments in detail in the next section.
-
+This will install a package into the conda base environment. 
+We will explain environments in detail in the next section.
 To search for a package, e.g. to find the correct name in conda or if it exists
 at all, issue the command:
 
 ```bash
-med0127:~$ conda search <string>
+hpc-cpu-123:~$ conda search <string>
 ```
 
 To choose a specific version (conda will install the latest version that is
 compatible with the current installed Python version), you can provide the
 version as follows:
 
 ```bash
-med0127:~$ conda install <package>=<version>
+hpc-cpu-123:~$ conda install <package>=<version>
 ```
 
 Please note that new conda installs may ship with a recently update Python version and not all packages might have been adapted.
@@ -102,13 +86,13 @@ E.g., if you find out that some packages don't work after starting out/upgrading
     Simply run
 
     ```bash
-    med0127:~$ conda install mamba
+    hpc-cpu-123:~$ conda install mamba
     ```
 
     With that, you can install software into your environment using the same syntax as for Conda:
 
     ```bash
-    med0127:~$ mamba install <package>
+    hpc-cpu-123:~$ mamba install <package>
     ```
 
 ## Creating an environment
@@ -125,18 +109,18 @@ environment, is is available in all other environments.
 To create a Python 2.7 environment and activate it, issue the following commands:
 
 ```bash
-med0127:~$ conda create -n py27 python=2.7
-med0127:~$ source activate py27
-(py27) med0127:~$
+hpc-cpu-123:~$ conda create -n py27 python=2.7
+hpc-cpu-123:~$ source activate py27
+(py27) hpc-cpu-123:~$
 ```
 
 From now on, conda will install packages into the `py27` environment when you issue
 the `install` command. To switch back to the root environment, simply deactivate the
 `py27` environment:
 
 ```bash
-(py27) med0127:~$ source deactivate py27
-med0127:~$
+(py27) hpc-cpu-123:~$ source deactivate py27
+hpc-cpu-123:~$
 ```
 
 But of course, as Python 2.7 is not supported any more by the Python Software Foundation, you should switch over to Python 3 already!

bih-cluster/docs/connecting/connecting.md

@@ -23,6 +23,7 @@ Follow these steps to connect to BIH HPC via the command line:
     # Charite Users
     $ ssh [email protected]
     $ ssh [email protected]
+
     # MDC Users
     $ ssh [email protected]
     $ ssh [email protected]
@@ -37,8 +38,15 @@ Follow these steps to connect to BIH HPC via the command line:
     Please also read [Advanced SSH](./advanced-ssh/overview.md) for more custom scenarios how to connect to BIH HPC.
     If you are using a Windows PC to access BIH HPC, please read [Connecting via SSH on Windows](./connecting-windows.md)
 
-5. Bonus: [Configure your SSH client :wrench: on Linux and Mac](advanced-ssh/linux.md) or [Windows](advanced-ssh/windows.md).
-6. Bonus: [Connect from external networks :flying_saucer:](./from-external.md).
+5. Allocate resources on a computation node using [Slurm](../slurm/overview.md). Do not compute on the login node!
+
+    ```bash
+    # Start interactive shell on computation node
+    $ srun --pty bash -i
+    ```
+
+6. Bonus: [Configure your SSH client :wrench: on Linux and Mac](advanced-ssh/linux.md) or [Windows](advanced-ssh/windows.md).
+7. Bonus: [Connect from external networks :flying_saucer:](./from-external.md).
 
 !!! tip "tl;dr"
 
@@ -49,10 +57,11 @@ Follow these steps to connect to BIH HPC via the command line:
         # Interactive login (choose one)
         ssh [email protected]
         ssh [email protected]
+        srun --pty bash -i
 
         # File Transfer (choose one)
-        sftp [email protected]
-        sftp [email protected]
+        sftp local/file [email protected]:remote/file
+        sftp [email protected]:remote/file local/file
 
         # Interactive login into the transfer nodes (choose one)
         ssh [email protected]

bih-cluster/docs/connecting/from-external.md

@@ -48,17 +48,17 @@ $ ssh [email protected]
 ```
 
 ## Charité Users
-You will then have to apply for (1) general VPN access and (2) extended VPN access to BIH HPC.
-Finally, you will be able to connect to BIH HPC from VPN.
+Access to BIH HPC from external networks (including Eduroam) requires a Charité VPN connection with special access permissions.
 
-### General Charite VPN Access
-You need to apply for Charite VPN access, if you haven't done so already.
+### General Charité VPN Access
+You need to apply for general Charité VPN access if you haven't done so already.
 The form can be found in the [Charite Intranet](https://intranet.charite.de/fileadmin/user_upload/portal/service/service_06_geschaeftsbereiche/service_06_14_it/VPN-Antrag_Mitarb_Stud.pdf) and contains further instructions.
+[Charité IT Helpdesk](mailto:[email protected]) can help you with any questions.
 
-### Zusatzantrag B (Recommended)
-You can find [Zusatzantrag B](https://intranet.charite.de/fileadmin/user_upload/portal/service/service_06_geschaeftsbereiche/service_06_14_it/VPN-Zusatzantrag_B.pdf) in the Charite intranet.
-Fill it out and ship it in addition to the general VPN access form from above.
-[Charite Helpdesk](mailto:[email protected]) can help you with any questions.
+### Zusatzantrag B
+Special permissions form B is also required for HPC access.
+You can find [Zusatzantrag B](https://intranet.charite.de/fileadmin/user_upload/portal/service/service_06_geschaeftsbereiche/service_06_14_it/VPN-Zusatzantrag_B.pdf) in the Charité intranet.
+Fill it out and send it to the same address as the general VPN access form above.
 
 Once you have been granted VPN access, start the client and connect to VPN.
 You will then be able to connect from your client in the VPN just as you do from your workstation.
@@ -67,9 +67,9 @@ You will then be able to connect from your client in the VPN just as you do from
 $ ssh [email protected]
 ```
 
-### Charite VDI
-Alternative to using Zusatzantrag B, you can also get access to the Charite VDI (Virtual Desktop Infrastructure).
-Here, you connect to a virtual desktop computer which is in the Charite network.
+### Charité VDI (Not recommended)
+Alternative to using Zusatzantrag B, you can also get access to the Charité VDI (Virtual Desktop Infrastructure).
+Here, you connect to a virtual desktop computer which is in the Charité network.
 From there, you can connect to the BIH HPC system.
 
 You need to apply for extended VPN access to be able to access the BIH VDI.

bih-cluster/docs/help/faq.md

@@ -1,25 +1,27 @@
 # Frequently Asked Questions
 
-## What is this Website?
-
-This is the BIH cluster documentation that was created and is maintained by BIH Core Unit Bioinformatics (CUBI) and BIH HPC IT with contributions by BIH HPC Users.
-The aim is to gather the information for using the cluster efficiently and helping common issues andproblems.
-
 ## Where can I get help?
-
 - Talk to your colleagues!
 - Have a look at our forums at [HPC-talk](https://hpc-talk.cubi.bihealth.org/) to see if someone already solved the same problem.
   If not, create a new topic. Administrators, CUBI, and other users can see and answer your question.
 - For problems while connecting and logging in, please contact [[email protected]](mailto:[email protected]) or [[email protected]](mailto:[email protected]).
 - For problems with BIH HPC please contact [[email protected]].
 
 ## I cannot connect to the cluster. What's wrong?
-
 Please see the section [Connection Problems](../connecting/connection-problems.md).
 
-## I'd like to learn more about Slurm
+## Connecting to the cluster takes a long time.
+The most probable cause for this is a conda installation which defaults to loading the _(Base)_ environment on login.
+To disable this behaviour you can run:
 
-- Some documentation is available on this website, e.g., start at [Slurm Quickstart](../slurm/quickstart.md).
+```sh
+$ conda config --set auto_activate_base false
+```
+
+You can also run the bash shell in verbose mode to find out exactly which command is slowing down login:
+```sh
+$ ssh [email protected] bash -iv
+```
 
 ## What is the difference between MAX and BIH cluster? What is their relation?
 
@@ -317,12 +319,7 @@ This is probably answered by the answer to [My jobs don't run in the partition I
 
 You cannot.
 
-## Why can I not mount a network volume from elsewhere on the cluster?
-
-For performance and stability reasons.
-Network volumes are notorious for degrading performance, depending on the used protocol, even stability.
-
-## How can I then access the files from my workstation/server?
+## How can I make workstation/server files available to the HPC?
 
 You can transfer files to the cluster through Rsync over SSH or through SFTP to the `hpc-transfer-1` or `hpc-transfer-2` node.
 
@@ -337,11 +334,6 @@ E.g., use the `-march=sandybridge` argument to the GCC/LLVM compiler executables
 If you absolutely need it, there are some boxes with more recent processors in the cluster (e.g., Haswell architecture).
 Look at the `/proc/cpuinfo` files for details.
 
-## Where should my (Mini)conda install go?
-
-As conda installations are big and contain many files, they should go into your `work` directory.
-**E.g., `/fast/users/$USER/work/miniconda` is appropriate.**
-
 ## I have problems connecting to the GPU node! What's wrong?
 
 Please check whether there might be other jobs waiting in front of you!
@@ -414,7 +406,6 @@ You can see the assignment of architectures to nodes using the `sinfo -o "%8P %.
 This will also display node partition, availability etc.
 
 ## Help, I'm getting a Quota Warning Email!
-
 No worries!
 
 As documented in the [Storage Locations](../storage/storage-locations.md) section, each user/project/group has three storage volumes:
@@ -433,22 +424,10 @@ Use the following command to list **all** files and directories in your home:
 hpc-login-1:~$ ls -la ~/
 ```
 
-You can use the following command to see how space each item takes up, including the hidden directories
-
-```bash
-hpc-login-1:~$ du -shc ~/.* ~/* --exclude=.. --exclude=.
-```
-
-In the case that, e.g., the `.cpan` directory is large, you can move it to `work` and create a symlink in its original place.
-
-```bash
-hpc-login-1:~$ mv ~/.cpan ~/work/.cpan
-hpc-login-1:~$ ln -sr ~/work/.cpan ~/.cpan
-```
+For more information on how to keep your home directory clean and avoid quota warnings, please read [Home Folder Quota](../storage/home-quota.md).
 
 ## I'm getting a "Disk quota exceeded" error.
-
-Most probably you are running into the same problem as described and solved in the entry [Help, I'm getting a Quota Warning Email!](#help-im-getting-a-quota-warning-email)
+Most probably you are running into the same problem as described above: [Help, I'm getting a Quota Warning Email!](#help-im-getting-a-quota-warning-email)
 
 ## Environment modules don't work and I get "module: command not found"
 
@@ -637,14 +616,6 @@ slurmstepd: error: task[0] unable to set taskset '0x0'
 This is a minor failure related to Slurm and cgroups.
 Your job **should** run through successfully despite this error (that is more of a warning for end-users).
 
-## My login stalls / something weird is happening
-
-You can try to run `ssh -l USER hpc-login-1.cubi.bihealth.org bash -iv`.
-This will run `bash -iv` instead of the normal login shell.
-The parameter `-i` is creating an interactive shell (which is what you want) and `-v` to see every command that is executed.
-This way you will see **every command** that is executed.
-You will also be able to identify at which point there is any stalling (e.g., activating conda via `source .../conda` when the fiel system is slow).
-
 ## How can I share files/collaborate with users from another work group?
 
 Please use [projects as documented here](../admin/getting-access.md#projects).

bih-cluster/docs/help/hpc-talk.md

@@ -1,13 +1,12 @@
 # HPC Talk
 
-Another community-driven possibility to get help is our HPC Talk portal. After this manual, it should be the first place to consult.
+Another community-driven possibility to get help is our “HPC Talk” forum. After this manual, it should be the first place to consult.
 
 https://hpc-talk.cubi.bihealth.org/
 
-For those who are familiar with it, it resembles the concept of stack overflow, only without the voting.
+Its main purpose is to serve as a FAQ, so with time and more people participating, you will more likely find an answer to your question.
+We also use it to make announcements and give an up-to-date status of current problems with the cluster, so it is worth logging in every once in a while.
+It is also a great first place to look at if you're experiencing problems with the cluster.
+Maybe it's a known issue.
 
-Its main purpose is to server as a FAQ, so with time and more people participating, you will more likely find an answer to your question.
-
-Also, we use it to make announcements and give an up-to-date status of current problems with the cluster, so it is worth to look at it every once in a while, or the first place to look at if you experience problems with the cluster to see if this is a known issue.
-
-Despite users also being able to anser questions, our admins do participate on a regular basis.
+Despite users also being able to answer questions, our admins do participate on a regular basis.

bih-cluster/docs/index.md

@@ -19,11 +19,9 @@ Read the following set of pages (in order) to learn how to get access and connec
 1. [Getting Access](admin/getting-access.md)
 2. [Connecting](connecting/connecting.md)
 3. [Storage](storage/storage-locations.md)
-4. [Getting Help](help/hpc-talk.md) ([Writing Good Tickets](help/good-tickets.md); if no answer found, contact the [HPC Helpdesk](help/helpdesk.md)).
-5. [HPC Tutorial](hpc-tutorial/episode-0.md)
-
-Then, continue reading through the manual.
-
+5. [Slurm](slurm/overview.md)
+6. [Getting Help](help/hpc-talk.md) ([Writing Good Tickets](help/good-tickets.md); if no answer found, contact the [HPC Helpdesk](help/helpdesk.md)).
+7. [HPC Tutorial](hpc-tutorial/episode-0.md)
 
 !!! note "Acknowledging BIH HPC Usage"
     Acknowledge usage of the cluster in your manuscript as *"Computation has been performed on the HPC for Research/Clinic cluster of the Berlin Institute of Health"*.