diff --git a/src/docs/storage/index.md b/src/docs/storage/index.md index 6bef605ed..d4c0344a8 100644 --- a/src/docs/storage/index.md +++ b/src/docs/storage/index.md @@ -1,7 +1,7 @@ # Storage on Sherlock Sherlock provides access to several file systems, each with distinct storage -characteristics. Each user and PI group get access to a set of pre-defined +characteristics. Each user and PI group get access to a set of predefined directories in these file systems to store their data. !!! danger "Sherlock is a compute cluster, not a storage system" @@ -26,7 +26,7 @@ and for some of them, purge policies (time-residency limits). | Name | Type | Backups / Snapshots | Performance | Purpose | Cost | | ------------------------ |--------------------- |------------------ | ----------- | ------- | ---- | -|`$HOME`, `$GROUP_HOME` | [NFS][url_NFS] | :fontawesome-solid-check:{: .chk_yes :} / :fontawesome-solid-check:{: .chk_yes :} | low | small, important files (source code, executables, configuration files...) | free | +|`$HOME`, `$GROUP_HOME` | [NFS][url_NFS] | :fontawesome-solid-check:{: .chk_yes :} / :fontawesome-solid-check:{: .chk_yes :} | low | small, important files (source code, executable files, configuration files...) | free | |`$SCRATCH`, `$GROUP_SCRATCH` | [Lustre][url_lustre] | :fontawesome-solid-xmark:{: .chk_no :} / :fontawesome-solid-xmark:{: .chk_no :} | high bandwidth | large, temporary files (checkpoints, raw application output...) | free | |`$L_SCRATCH` | local SSD | :fontawesome-solid-xmark:{: .chk_no :} / :fontawesome-solid-xmark:{: .chk_no :} | low latency, high IOPS | job specific output requiring high IOPS | free | |`$OAK` | [Lustre][url_lustre] | option / :fontawesome-solid-xmark:{: .chk_no :} | moderate | long term storage of research data | volume-based[^oak_sd] | @@ -86,12 +86,12 @@ Retention types: * **job lifetime**: files are only kept for the duration of the job and are automatically purged when the job ends. -!!! info "Global failsafe user and quota groups on `/scratch`" +!!! info "Global fail-safe user and quota groups on `/scratch`" To prevent potential issues which would result in the file system filling up completely and making it unusable for everyone, additional user and group-level quotas are in place on the `/scratch` file system, as a - failsafe: + fail-safe: * a user will not be able to use more than 250 TB (50M inodes) in total, in all the `/scratch` directories they have access to. @@ -176,9 +176,59 @@ $ sh_quota -f SCRATCH -j } ``` + +#### Locating large directories + +It's not always easy to identify files and directories that take the most space +when getting close to the quota limits. Some tools can help with that. + +* [`du`][url_du] can be used to display the volume used by files and + directories, in a given folder: + + ``` none + $ cd mydir/ + $ du --human-readable --summarize * + 101M dir + 2.0M file + ``` + + !!! note + + `du` will ignore hidden entries (everything that starts with a dot (`.`)). + So when using it in your `$HOME` directory, it will skip things like + `.cache` or `.conda`, which can contain significant volumes. + + +* [`ncdu`][url_ncdu] is an interactive disk usage analyzer, that generates + visual representation of the volume (and inode count) for directories. To run + it, you need to load the `ncdu` module, and then run it on your directory of + choice: + + ``` none + $ ml system ncdu + $ ncdu $HOME + ``` + + For very large directories, running `ncdu` in an interactive shell on a + compute node is recommended, via [`sh_dev`][url_sh_dev]. + + You'll been there presented with an interactive file browser, showing + information about the volume used by your directories, which should make easy + to pinpoint where most space is used. + +!!! info + + Note that any tool you use to view directory contents will only be able to + show files that your user account has read access to. So on group-shared + spaces, if you see a major difference between the totals from a tool like + `ncdu` and the information reported by `sh_quota`, that can be an indicator + that one of your group members has restricted permissions on a large number + of items in your space. + + ## Where should I store my files? -!!! important "Not all filesystems are equivalent" +!!! warning "Not all filesystems are equivalent" Choosing the appropriate storage location for your files is an essential step towards making your utilization of the cluster the most efficient @@ -247,6 +297,9 @@ Transfer][url_data_sshfs] page. [url_oak]: //uit.stanford.edu/service/oak-storage [url_data_sshfs]: /docs/storage/data-transfer#sshfs [url_purge]: /docs/storage/filesystems/#expiration-policy +[url_du]: //www.gnu.org/software/coreutils/manual/html_node/du-invocation.html#du-invocation +[url_ncdu]: //dev.yorhel.nl/ncdu +[url_sh_dev]: /docs/user-guide/running-jobs/#interactive-jobs [comment]: # (footnotes -----------------------------------------------------)