Merge pull request #737 from SciLifeLab/dev

Prepare Release 2.3
SciLifeLab · Feb 28, 2019 · 7f7f2c2 · 7f7f2c2
2 parents 88921e6 + 38fad0a
commit 7f7f2c2
Show file tree

Hide file tree

Showing 57 changed files with 1,293 additions and 1,222 deletions.
diff --git a/.github/RELEASE_CHECKLIST.md b/.github/RELEASE_CHECKLIST.md
@@ -4,11 +4,11 @@
 
 1.  Check that everything is ready to go
 
-    -   [PRs](https://github.com/SciLifeLab/Sarek/pulls) are merged
+    -   Desired [PRs](https://github.com/SciLifeLab/Sarek/pulls) are merged
     -   [Travis tests](https://travis-ci.org/SciLifeLab/Sarek/branches) are passing on `dev`
 
 2.  Increase version number following [semantic versioning](http://semver.org/spec/v2.0.0.html)
-3.  Choose an appropriate codename for the release
+3.  Choose an appropriate codename for the release (if major or minor)
     -   i.e. Peaks in [Sarek National Park](https://en.wikipedia.org/wiki/Sarek_National_Park#Topography)
 4.  Build docker containers.
 
@@ -32,5 +32,8 @@
 13. Update [bio.tools](https://bio.tools/Sarek) with the new release details
 14. Tweet that a new version is released
 15. Add a new `Unreleased` section in `CHANGELOG.md` for the `dev` version
-16. Commit and push. Continue making more awesome :metal:
-17. Have fika :cake:
+16. Download all new containers to `/sw/data/uppnex/ToolBox/sarek` on `rackham`
+17. Download all new containers to `/btb/containers` on `munin`
+18. Update symbolic links to `latest`
+19. Commit and push. Continue making more awesome :metal:
+20. Have fika :cake:
diff --git a/.travis.yml b/.travis.yml
@@ -13,6 +13,7 @@ env:
   matrix:
     - TEST=SOMATIC
     - TEST=GERMLINE
+    - TEST=TARGETED
     - TEST=ANNOTATEVEP
     - TEST=ANNOTATESNPEFF
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -5,6 +5,64 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).
 
+## [2.3] - Äpar - 2019-02-27
+
+### `Added`
+-   [#628](https://github.com/SciLifeLab/Sarek/pull/628), [#722](https://github.com/SciLifeLab/Sarek/pull/722) - `ASCAT` now use `.gc` file
+-   [#712](https://github.com/SciLifeLab/Sarek/pull/712), [#718](https://github.com/SciLifeLab/Sarek/pull/718) - Added possibilities to run Sarek with `conda`
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - Annotation documentation
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - Helper script to download `snpeff` and `VEP` cache files
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - New `--annotation_cache`, `--snpEff_cache`, `--vep_cache` parameters
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - Possibility to use cache wen annotating with `snpEff` and `VEP`
+-   [#722](https://github.com/SciLifeLab/Sarek/pull/722) - Add path to ASCAT `.gc` file in `igenomes.config`
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Update `Sarek-data` submodule with multiple patients TSV file
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Add `cadd_WG_SNVs`, `cadd_WG_SNVs_tbi`, `cadd_InDels`, `cadd_InDels_tbi` and `cadd_cache` params
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Add tabix indexed cache for VEP
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - New `DownloadCADD` process to download CADD files
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Specify values for `cadd_WG_SNVs`, `cadd_WG_SNVs_tbi`, `cadd_InDels`, `cadd_InDels_tbi` and `cadd_cache` params in `munin.conf` file
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Use `cadd_cache` param for optional use of CADD VEP plugin in `annotate.nf`
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - VEP cache has now fasta files for `--HGVS`
+-   [#735](https://github.com/SciLifeLab/Sarek/pull/735) - Added `--exome` for Manta, and for StrelkaBP
+-   [#735](https://github.com/SciLifeLab/Sarek/pull/735) - Added Travis CI test for targeted
+
+### `Changed`
+
+-   [#710](https://github.com/SciLifeLab/Sarek/pull/710) - Improve release checklist and script
+-   [#711](https://github.com/SciLifeLab/Sarek/pull/711) - Improve configuration priorities
+-   [#716](https://github.com/SciLifeLab/Sarek/pull/716) - Update paths to containers and iGenomes
+-   [#717](https://github.com/SciLifeLab/Sarek/pull/717) - `checkFileExtension` has changed to `hasExtension`, and now only verify if file has extension
+-   [#717](https://github.com/SciLifeLab/Sarek/pull/717) - `fastqFiles` renamed to `inputFiles`
+-   [#717](https://github.com/SciLifeLab/Sarek/pull/717) - `mapping` step can now map BAM files too
+-   [#717](https://github.com/SciLifeLab/Sarek/pull/717) - `MapReads` can now convert BAM to FASTQ and feed it to BWA on the fly
+-   [#717](https://github.com/SciLifeLab/Sarek/pull/717), [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Update documentation
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - `snpeff` and `vep` containers are now built with conda
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - `vepCacheVersion` is now defined in `conf/genomes.config` or `conf/igenomes.config`
+-   [#722](https://github.com/SciLifeLab/Sarek/pull/722) - Add path to ASCAT `.gc` file in `igenomes.config`
+-   [#722](https://github.com/SciLifeLab/Sarek/pull/722) - Update `Sarek-data` submodule
+-   [#723](https://github.com/SciLifeLab/Sarek/pull/723), [#725](https://github.com/SciLifeLab/Sarek/pull/725) - Update docs
+-   [#724](https://github.com/SciLifeLab/Sarek/pull/724) - Improved AwsBatch configuration
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Improved usage of `targetBED` params
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Strelka Best Practices output is now prefixed with `StrelkaBP_`
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - VCFs and Annotated VCFs are now ordered by Patient, then tools
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Merge `buildContainers.nf` and `buildReferences.nf` in `build.nf`
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Reduce number of CPUs for `RunVEP` to `4` cf: [VEP docs](https://www.ensembl.org/info/docs/tools/vep/script/vep_other.html)
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Update VEP from `95.1` to `95.2`
+
+### `Removed`
+-   [#715](https://github.com/SciLifeLab/Sarek/pull/715) - Remove `defReferencesFiles` function from `buildReferences.nf`
+-   [#719](https://github.com/SciLifeLab/Sarek/pull/719) - `snpEff` base container is no longer used
+-   [#721](https://github.com/SciLifeLab/Sarek/pull/721) - Remove COSMIC docs
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Remove `defineDirectoryMap()`
+-   [#732](https://github.com/SciLifeLab/Sarek/pull/732) - Removed `--database` option for VEP cf: [VEP docs](https://www.ensembl.org/info/docs/tools/vep/script/vep_other.html)
+
+### `Fixed`
+-   [#720](https://github.com/SciLifeLab/Sarek/pull/720) - bamQC is now run on the recalibrated bams, and not after MarkDuplicates
+-   [#726](https://github.com/SciLifeLab/Sarek/pull/726) - Fix Ascat ref file input (one file can't be a set)
+-   [#727](https://github.com/SciLifeLab/Sarek/pull/727) - bamQC outputs are no longer overwritten (name of dir is now the file instead of sample)
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Fix issue with annotation that was consuming `cache` channels
+-   [#728](https://github.com/SciLifeLab/Sarek/pull/728) - Fix multi sample TSV file [#691](https://github.com/SciLifeLab/Sarek/issues/691)
+-   [#733](https://github.com/SciLifeLab/Sarek/pull/733) - Fix the possibility to specify reference files on the command line
+
 ## [2.2.2] - 2018-12-19
 
 ### `Added`
@@ -19,10 +77,10 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### `Changed`
 
--   [#678](https://github.com/SciLifeLab/Sarek/pull/678) - Changing VEP to v92 and adjusting CPUs for VEP
 -   [#663](https://github.com/SciLifeLab/Sarek/pull/663) - Update `do_release.sh` script
 -   [#671](https://github.com/SciLifeLab/Sarek/pull/671) - publishDir modes are now params
 -   [#677](https://github.com/SciLifeLab/Sarek/pull/677), [#698](https://github.com/SciLifeLab/Sarek/pull/698), [#703](https://github.com/SciLifeLab/Sarek/pull/703) - Update docs
+-   [#678](https://github.com/SciLifeLab/Sarek/pull/678) - Changing VEP to v92 and adjusting CPUs for VEP
 -   [#679](https://github.com/SciLifeLab/Sarek/pull/679) - Update old awsbatch configuration
 -   [#682](https://github.com/SciLifeLab/Sarek/pull/682) - Specifications for memory and cpus for awsbatch
 -   [#693](https://github.com/SciLifeLab/Sarek/pull/693) - Qualimap bamQC is now ran after mapping and after recalibration for better QC
@@ -63,15 +121,15 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 -   [#616](https://github.com/SciLifeLab/Sarek/pull/616) - Update documentation
 -   [#620](https://github.com/SciLifeLab/Sarek/pull/620) - Add `tmp/` to `.gitignore`
 -   [#625](https://github.com/SciLifeLab/Sarek/pull/625) - Add [`pathfindr`](https://github.com/NBISweden/pathfindr) as a submodule
--   [#639](https://github.com/SciLifeLab/Sarek/pull/639) - Add a complete example analysis to docs
 -   [#635](https://github.com/SciLifeLab/Sarek/pull/635) - To process targeted sequencing with a target BED
+-   [#639](https://github.com/SciLifeLab/Sarek/pull/639) - Add a complete example analysis to docs
 -   [#640](https://github.com/SciLifeLab/Sarek/pull/640), [#642](https://github.com/SciLifeLab/Sarek/pull/642) - Add helper script for changing version number
 
 ### `Changed`
 
 -   [#608](https://github.com/SciLifeLab/Sarek/pull/608) - Update Nextflow required version
--   [#616](https://github.com/SciLifeLab/Sarek/pull/616) - Update CHANGELOG
 -   [#615](https://github.com/SciLifeLab/Sarek/pull/615) - Use `splitCsv` instead of `readlines`
+-   [#616](https://github.com/SciLifeLab/Sarek/pull/616) - Update CHANGELOG
 -   [#621](https://github.com/SciLifeLab/Sarek/pull/621), [#638](https://github.com/SciLifeLab/Sarek/pull/638) - Improve install script
 -   [#621](https://github.com/SciLifeLab/Sarek/pull/621), [#638](https://github.com/SciLifeLab/Sarek/pull/638) - Simplify tests
 -   [#627](https://github.com/SciLifeLab/Sarek/pull/627), [#629](https://github.com/SciLifeLab/Sarek/pull/629), [#637](https://github.com/SciLifeLab/Sarek/pull/637) - Refactor docs
@@ -81,9 +139,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 -   [#638](https://github.com/SciLifeLab/Sarek/pull/638) - Use correct `.simg` extension for Singularity images
 -   [#639](https://github.com/SciLifeLab/Sarek/pull/639) - Smaller refactoring of the docs
 -   [#640](https://github.com/SciLifeLab/Sarek/pull/640) - Update RELEASE_CHECKLIST
--   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - Update conda channel order priorities
 -   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - MultiQC 1.5 -> 1.6
 -   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - Qualimap 2.2.2a -> 2.2.2b
+-   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - Update conda channel order priorities
 -   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - VCFanno 0.2.8 -> 0.3.0
 -   [#642](https://github.com/SciLifeLab/Sarek/pull/642) - VCFtools 0.1.15 -> 0.1.16
 
@@ -128,14 +186,14 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 -   [#582](https://github.com/SciLifeLab/Sarek/pull/582), [#587](https://github.com/SciLifeLab/Sarek/pull/587) - Update figures
 -   [#595](https://github.com/SciLifeLab/Sarek/pull/595) - Function `defineDirectoryMap()` is now part of `SarekUtils`
 -   [#595](https://github.com/SciLifeLab/Sarek/pull/595) - Process `GenerateMultiQCconfig` replace by function `createMultiQCconfig()`
--   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - Move `checkFileExtension()`, `checkParameterExistence()`, `checkParameterList()`, `checkReferenceMap()`, `checkRefExistence()`, `extractBams()`, `extractGenders()`, `returnFile()`, `returnStatus()` and `returnTSV()` functions to `SarekUtils`
 -   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - `extractBams()` now takes an extra parameter.
--   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - Replace depreciated operator `phase` by `join`.
+-   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - Move `checkFileExtension()`, `checkParameterExistence()`, `checkParameterList()`, `checkReferenceMap()`, `checkRefExistence()`, `extractBams()`, `extractGenders()`, `returnFile()`, `returnStatus()` and `returnTSV()` functions to `SarekUtils`
 -   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - Reduce data footprint for Process `CreateRecalibrationTable`
+-   [#597](https://github.com/SciLifeLab/Sarek/pull/597) - Replace depreciated operator `phase` by `join`.
 -   [#599](https://github.com/SciLifeLab/Sarek/pull/599) - Merge is tested with `ANNOTATEALL`
 -   [#604](https://github.com/SciLifeLab/Sarek/pull/604) - Synching `GRCh38` `wgs_calling_regions` bedfiles
--   [#607](https://github.com/SciLifeLab/Sarek/pull/607) - Update to GATK4
 -   [#607](https://github.com/SciLifeLab/Sarek/pull/607) - One container approach
+-   [#607](https://github.com/SciLifeLab/Sarek/pull/607) - Update to GATK4
 -   [#608](https://github.com/SciLifeLab/Sarek/pull/608) - Update Nextflow required version
 -   [#616](https://github.com/SciLifeLab/Sarek/pull/616) - Update CHANGELOG
 -   [#617](https://github.com/SciLifeLab/Sarek/pull/617) - Replace depreciated $name syntax with withName
@@ -205,8 +263,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### `Fixed`
 
--   [#533](https://github.com/SciLifeLab/Sarek/issues/533) - Replace `VEP` `--pick` option by `--per_gene`
 -   [#530](https://github.com/SciLifeLab/Sarek/issues/530) - use `$PWD` for default `outDir`
+-   [#533](https://github.com/SciLifeLab/Sarek/issues/533) - Replace `VEP` `--pick` option by `--per_gene`
 
 ## [1.2.5] - 2018-01-18
 
@@ -250,9 +308,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.
 
 ### `Fixed`
 
--   [#475](https://github.com/SciLifeLab/Sarek/issues/475) - 16 cpus for local executor
 -   [#357](https://github.com/SciLifeLab/Sarek/issues/357) - `ASCAT` works for GRCh38
 -   [#471](https://github.com/SciLifeLab/Sarek/issues/471) - Running `Singularity` on `/scratch`
+-   [#475](https://github.com/SciLifeLab/Sarek/issues/475) - 16 cpus for local executor
 -   [#480](https://github.com/SciLifeLab/Sarek/issues/480) - No `tsv` file needed for step `annotate`
 
 ## [1.2.2] - 2017-10-06

diff --git a/Dockerfile b/Dockerfile
@@ -1,10 +1,10 @@
 FROM nfcore/base:latest
 
 LABEL \
-  authors="Maxime.Garcia@scilifelab.se, Szilveszter.Juhos@scilifelab.se" \
+  authors="maxime.garcia@scilifelab.se, szilveszter.juhos@scilifelab.se" \
   description="Image with tools used in Sarek" \
-	maintainer="Maxime Garcia <[email protected]>, Szilveszter Juhos <Szilveszter.Juhos@scilifelab.se>"
+	maintainer="Maxime Garcia <[email protected]>, Szilveszter Juhos <szilveszter.juhos@scilifelab.se>"
 
 COPY environment.yml /
 RUN conda env create -f /environment.yml && conda clean -a
-ENV PATH /opt/conda/envs/sarek-2.2.2/bin:$PATH
+ENV PATH /opt/conda/envs/sarek-2.3/bin:$PATH
diff --git a/README.md b/README.md
@@ -82,13 +82,15 @@ The Sarek pipeline comes with documentation in the `docs/` directory:
 06. [Configuration and profiles documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/CONFIG.md)
 07. [Intervals documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/INTERVALS.md)
 08. [Running the pipeline](https://github.com/SciLifeLab/Sarek/blob/master/docs/USAGE.md)
-09. [Command line parameters](https://github.com/SciLifeLab/Sarek/blob/master/docs/PARAMETERS.md)
-10. [Examples](https://github.com/SciLifeLab/Sarek/blob/master/docs/USE_CASES.md)
-11. [Input files documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/INPUT.md)
-12. [Processes documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/PROCESS.md)
-13. [Documentation about containers](https://github.com/SciLifeLab/Sarek/blob/master/docs/CONTAINERS.md)
-14. [More information about ASCAT](https://github.com/SciLifeLab/Sarek/blob/master/docs/ASCAT.md)
-15. [Output documentation structure](https://github.com/SciLifeLab/Sarek/blob/master/docs/OUTPUT.md)
+09. [Running the pipeline using Conda](https://github.com/SciLifeLab/Sarek/blob/master/docs/CONDA.md)
+10. [Command line parameters](https://github.com/SciLifeLab/Sarek/blob/master/docs/PARAMETERS.md)
+11. [Examples](https://github.com/SciLifeLab/Sarek/blob/master/docs/USE_CASES.md)
+12. [Input files documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/INPUT.md)
+13. [Processes documentation](https://github.com/SciLifeLab/Sarek/blob/master/docs/PROCESS.md)
+14. [Documentation about containers](https://github.com/SciLifeLab/Sarek/blob/master/docs/CONTAINERS.md)
+15. [Complementary information about ASCAT](https://github.com/SciLifeLab/Sarek/blob/master/docs/ASCAT.md)
+16. [Complementary information about annotations](https://github.com/SciLifeLab/Sarek/blob/master/docs/ANNOTATION.md)
+17. [Output documentation structure](https://github.com/SciLifeLab/Sarek/blob/master/docs/OUTPUT.md)
 
 ## Contributions & Support
 
@@ -117,6 +119,7 @@ Helpful contributors:
 * [Björn Nystedt](https://github.com/bjornnystedt)
 * [Pall Olason](https://github.com/pallolason)
 * [Aron Skaftason](https://github.com/arontommi)
+* [Nilesh Tawari](https://github.com/nilesh-tawari)
 
 --------------------------------------------------------------------------------
 

diff --git a/Sarek-data b/Sarek-data
diff --git a/Singularity b/Singularity
@@ -4,10 +4,10 @@ Bootstrap:docker
 %labels
     MAINTAINER Maxime Garcia <[email protected]>
     DESCRIPTION Singularity image containing all requirements for the Sarek pipeline
-    VERSION 2.2.2
+    VERSION 2.3
 
 %environment
-    PATH=/opt/conda/envs/sarek-2.2.2/bin:$PATH
+    PATH=/opt/conda/envs/sarek-2.3/bin:$PATH
     export PATH
 
 %files
+587 −0		reference/1000G_phase3_20130502_SNP_maf0.3.small.loci.gc
+ −		reference/b37_cosmic_v54_120711.small.vcf.gz
+ −		reference/b37_cosmic_v74.noCHR.sort.4.1.small.vcf.gz
+5 −0		testdata/tsv/dream-test-normal-s3.tsv
+24 −0		testdata/tsv/tiny-multiple.tsv