change defaults for Mount Sinai

README.md

@@ -22,6 +22,17 @@
 
 ## Install
 
+To install the preconfigured Goate Lab settings, run the following:
+
+```bash
+mkdir -p ~/.config/snakemake && \
+  git clone -b preconfigured_goatelab \
+    https://github.com/BEFH/lsf.git \
+    ~/.config/snakemake/lsf
+```
+
+You can modify the presets in `~/.config/snakemake/lsf/config.yaml` and `~/.config/snakemake/lsf/CookieCutter.py`
+
 ### Dependencies
 
 This profile is deployed using [Cookiecutter][cookiecutter-repo]. If you do not have
@@ -53,9 +64,11 @@ You will then be prompted to set some default parameters.
 
 #### `LSF_UNIT_FOR_LIMITS`
 
-**Default**: `KB`  
+**Default**: `KB`
 **Valid options:** `KB`, `MB`, `GB`, `TB`, `PB`, `EB`, `ZB`
 
+***On Minerva, this is in MB.***
+
 **⚠️IMPORTANT⚠️**: This **must** be set to the same as [`LSF_UNIT_FOR_LIMITS`][limits] on
 your cluster. This value is stored in your cluster's [`lsf.conf`][lsf-conf] file. In
 general, this file is located at `${LSF_ENVDIR}/lsf.conf`. So the easiest way to get
@@ -77,7 +90,7 @@ submitting jobs. See [here][18] for further information.
 
 #### `UNKWN_behaviour`
 
-**Default**: `wait`  
+**Default**: `wait`
 **Valid options**: `wait`, `kill`
 
 When LSF returns a job status of `UNKWN` do you want to wait for the host the job is
@@ -86,7 +99,7 @@ outlined [here][job_kill]?
 
 #### `ZOMBI_behaviour`
 
-**Default**: `ignore`  
+**Default**: `ignore`
 **Valid options**: `ignore`, `kill`
 
 When LSF returns a job status of `ZOMBI` do you want to ignore this (not clean it up) or
@@ -95,9 +108,9 @@ considered failed.
 
 #### `latency_wait`
 
-**Default:** `5`
+**Default:** `10`
 
-This sets the default `--latency-wait/--output-wait/-w` parameter in `snakemake`.  
+This sets the default `--latency-wait/--output-wait/-w` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -109,10 +122,10 @@ From the `snakemake --help` menu
 
 #### `use_conda`
 
-**Default**: `False`  
+**Default**: `True`
 **Valid options:** `False`, `True`
 
-This sets the default `--use-conda` parameter in `snakemake`.  
+This sets the default `--use-conda` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -123,10 +136,10 @@ From the `snakemake --help` menu
 
 #### `use_singularity`
 
-**Default**: `False`  
+**Default**: `True`
 **Valid options:** `False`, `True`
 
-This sets the default `--use-singularity` parameter in `snakemake`.  
+This sets the default `--use-singularity` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -139,7 +152,7 @@ From the `snakemake --help` menu
 
 **Default**: `0`
 
-This sets the default `--restart-times` parameter in `snakemake`.  
+This sets the default `--restart-times` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -150,10 +163,10 @@ From the `snakemake --help` menu
 
 #### `print_shell_commands`
 
-**Default**: `False`  
+**Default**: `True`
 **Valid options:** `False`, `True`
 
-This sets the default ` --printshellcmds/-p` parameter in `snakemake`.  
+This sets the default ` --printshellcmds/-p` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -162,9 +175,15 @@ From the `snakemake --help` menu
 
 #### `jobs`
 
-**Default**: `500`
+Determine how many jobs are supported by running
 
-This sets the default `--cores/--jobs/-j` parameter in `snakemake`.  
+```
+ulimit -u
+```
+
+**Default**: `2000`
+
+This sets the default `--cores/--jobs/-j` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -179,7 +198,7 @@ the same time<sup>[1][1]</sup>.
 
 #### `default_mem_mb`
 
-**Default**: `1024`
+**Default**: `4096`
 
 This sets the default memory, in megabytes, for a `rule` being submitted to the cluster
 without `mem_mb` set under `resources`.
@@ -196,13 +215,13 @@ to the working directory of the pipeline. If it does not exist, it will be creat
 
 The log files for a given rule are organised into sub-directories. This is to avoid
 having potentially thousands of files in one directory, as this can cause file system
-issues.  
+issues.
 If you want to find the log files for a rule called `foo`, with wildcards
 `sample=a,ext=fq` then this would be located at
 `logs/cluster/foo/sample=a,ext=fq/jobid<jobid>-<uuid>.out` for the [standard
-output][bsub-o] and with extension `.err` for the [standard error][bsub-e].  
+output][bsub-o] and with extension `.err` for the [standard error][bsub-e].
 `<jobid>` is the internal jobid used by `snakemake` and is the same across multiple
-attempts at running the same rule.  
+attempts at running the same rule.
 [`<uuid>`][uuid] is a random 28-digit, separated by `-`, and is specific to each attempt
 at running a rule. So if a rule fails, and is restarted, the uuid will be different.
 
@@ -216,7 +235,7 @@ specific rule by following the instructions
 **Default**: None
 
 The default queue on the cluster to submit jobs to. If left unset, then the default on
-your cluster will be used.  
+your cluster will be used.
 The `bsub` parameter that this controls is [`-q`][bsub-q].
 
 #### `default_project`
@@ -230,9 +249,9 @@ The `bsub` parameter that this controls is [`-P`][bsub-P].
 
 #### `max_status_checks_per_second`
 
-**Default**: `10`
+**Default**: `5`
 
-This sets the default `--max-status-checks-per-second` parameter in `snakemake`.  
+This sets the default `--max-status-checks-per-second` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -243,9 +262,9 @@ From the `snakemake --help` menu
 
 #### `max_jobs_per_second`
 
-**Default**: `10`
+**Default**: `5`
 
-This sets the default `--max-jobs-per-second` parameter in `snakemake`.  
+This sets the default `--max-jobs-per-second` parameter in `snakemake`.
 From the `snakemake --help` menu
 
 ```text
@@ -259,7 +278,7 @@ From the `snakemake --help` menu
 **Default**: `lsf`
 
 The name to use for this profile. The directory for the profile is created as this name
-i.e. `$HOME/.config/snakemake/<profile_name>`.  
+i.e. `$HOME/.config/snakemake/<profile_name>`.
 This is also the value you pass to `snakemake --profile <profile_name>`.
 
 ## Usage
@@ -313,7 +332,7 @@ rule foo:
     output: "bar.txt"
     shell:
         "grep 'bar' {input} > {output}"
-        
+
 rule bar:
     input: "bar.txt"
     output: "file.out"
@@ -334,7 +353,7 @@ foo:
 ```
 
 In this example, we specify a default (`__default__`) [project][bsub-P] (`-P`) and
-[runtime limit][bsub-W] (`-W`) that will apply to all rules.  
+[runtime limit][bsub-W] (`-W`) that will apply to all rules.
 We then override the project and, additionally, specify [GPU resources][bsub-gpu] for
 the rule `foo`.
 
@@ -380,7 +399,7 @@ complex log file naming scheme. As the status-checker uses `tail` to get the sta
 the standard output log file of the job is very large, then status checking will be
 slowed down as a result. If you run into these problems and the `tail` solution is no
 feasible, the first suggestion would be to reduce `--max_status_checks_per_second` and
-see if this helps.  
+see if this helps.
 Please raise an issue if you experience this, and the log file check doesn't seem to
 work.
 
@@ -411,4 +430,3 @@ Please refer to [`CONTRIBUTING.md`](CONTRIBUTING.md).
 [yaml-collections]: https://yaml.org/spec/1.2/spec.html#id2759963
 [leandro]: https://github.com/leoisl
 [snakemake_params]: https://snakemake.readthedocs.io/en/stable/executable.html#all-options
-

cookiecutter.json

@@ -1,18 +1,18 @@
 {
-  "LSF_UNIT_FOR_LIMITS": ["KB", "MB", "GB", "TB", "PB", "EB", "ZB"],
+  "LSF_UNIT_FOR_LIMITS": "MB",
   "UNKWN_behaviour": ["wait", "kill"],
   "ZOMBI_behaviour": ["ignore", "kill"],
-  "latency_wait": 5,
-  "use_conda": false,
-  "use_singularity": false,
+  "latency_wait": 10,
+  "use_conda": true,
+  "use_singularity": true,
   "restart_times": 0,
-  "print_shell_commands": false,
-  "jobs": 500,
-  "default_mem_mb": 1024,
+  "print_shell_commands": true,
+  "jobs": 2000,
+  "default_mem_mb": 4096,
   "default_cluster_logdir": "logs/cluster",
   "default_queue": "",
   "default_project": "",
-  "max_status_checks_per_second": 10,
-  "max_jobs_per_second": 10,
+  "max_status_checks_per_second": 5,
+  "max_jobs_per_second": 5,
   "profile_name": "lsf"
 }