From 2eec965e450ec057348fce3133a294262ba08b5b Mon Sep 17 00:00:00 2001 From: allanbolipata <33229763+allanbolipata@users.noreply.github.com> Date: Wed, 22 May 2019 09:44:01 -0400 Subject: [PATCH 1/6] Small changes to the README.md --- README.md | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 85aaedb33..b72641675 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ https://www.nextflow.io/ Inspiration from `Sarek` at the NBIS and SciLifeLab in Sweden: https://github.com/SciLifeLab/Sarek -Vaporware contains several pipeline scripts (`pipeline.nf`, `make_bam_and_qc.nf`, `somatic.nf`, `germline.nf`). A detalied explanation for each step is provided below. +Vaporware contains several pipeline `*.nf` scripts but all functionality is contained in `pipeline.nf`. ## Setup instructions @@ -29,6 +29,9 @@ This installs Nextflow within the subdirectory `vaporware`. Optionally, move the `nextflow` file to a directory accessible by your `$PATH` variable in order to avoid remembering and typing the full path to `nextflow` each time. +## Description of pipeline.nf + +The `pipeline.nf` script takes a mapping and pairing file to perform BAM preprocessing, somatic analysis, and germline analysis. BAM Preprocessing always occurs; to activate the somatic and germline pipelines, use the `--somatic` and `--germline`, respectively, at the command line. See the sections below for command examples. ## For submitting via LSF on juno @@ -71,7 +74,7 @@ nextflow run pipeline.nf --somatic --germline --mapping test_inputs/aws/test_mak **NOTE:** To being able to run locally you need to provide reference files from `conf/references.config` and create `samples.tsv` as described in the wiki page [Bioinformatic components](https://github.com/mskcc/vaporware/wiki/Bioinformatic-Components) -The default parameters are for local-use *WITHOUT* containers +The default parameters are for local-use *WITHOUT* containers. * For Docker use, do the following: @@ -85,6 +88,16 @@ nextflow run pipeline.nf --somatic --germline --mapping mapping.tsv --pairing pa nextflow run pipeline.nf --somatic --germline --mapping mapping.tsv --pairing pairing.tsv -profile singularity ``` +## Useful Tips + +* __Number of dashes matter__: Nextflow has a quirk where Nextflow executor-specific, built-in flags are initiated with a single dash, whereas parameters we define require two. For example, to define which run profile to use, we call Nextflow's built-in feature `-profile`; to do resume, we use `-resume`. To provide mapping or pairing input file paths, which call functions the development team at MSKCC had to write, we have to call it with `--mapping` and `--pairing`, respectively. + +* __Modify and Resume__: Nextflow supports [modify and resume](https://www.nextflow.io/docs/latest/getstarted.html?#modify-and-resume); to activate this feature, use `-resume` (note the single dash) at the command line to use Nextflow's cache history and continue a job from where it left off. This even lets you make changes to values in the script and continue from there - Nextflow will use the cached information from the unchanged sections while running only the modified processes. + +* __View the Nextflow Log__: You can access the Nextflow cache metadata by running `nextflow log`. It contains information like TIMESTAMP, DURATION, and RUN_NAME, and a STATUS indicating a failed or successful run, among others. The values under RUN_NAME can also be submitted following the `-resume` flag to resume previously-run Nextflow jobs. + +* __Run/Skip specific tools:__ `pipeline.nf` lets you run only specific tools you want and skip others. The aforementioned `--somatic` and `--germline` flags already have a preset of tools to run, but you can limit this further by providing the `--tools` flag, followed by a comma-delimited string. For example, to use only DELLY for your somatic/germline runs, do `--somatic --germline --tools delly`; to use MuTect2, Manta, and Strelka2, do `--somatic --tools mutect2,manta,strelka2`. + ## Components The following links to the wiki provide greater details: From 09370291ce2467ef326847928a957b5f6d646a52 Mon Sep 17 00:00:00 2001 From: Allan Bolipata Date: Wed, 22 May 2019 11:08:55 -0400 Subject: [PATCH 2/6] Changed some awkward verbage --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index b72641675..2b9e8eb05 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ nextflow run pipeline.nf --somatic --germline --mapping mapping.tsv --pairing pa * __View the Nextflow Log__: You can access the Nextflow cache metadata by running `nextflow log`. It contains information like TIMESTAMP, DURATION, and RUN_NAME, and a STATUS indicating a failed or successful run, among others. The values under RUN_NAME can also be submitted following the `-resume` flag to resume previously-run Nextflow jobs. -* __Run/Skip specific tools:__ `pipeline.nf` lets you run only specific tools you want and skip others. The aforementioned `--somatic` and `--germline` flags already have a preset of tools to run, but you can limit this further by providing the `--tools` flag, followed by a comma-delimited string. For example, to use only DELLY for your somatic/germline runs, do `--somatic --germline --tools delly`; to use MuTect2, Manta, and Strelka2, do `--somatic --tools mutect2,manta,strelka2`. +* __Run/Skip specific tools:__ `pipeline.nf` lets you run tools you want and skip others. The aforementioned `--somatic` and `--germline` flags already have a preset of tools to include during a run, but you can limit this further by providing the `--tools` flag, followed by a comma-delimited string. For example, to use only DELLY for your somatic/germline runs, do `--somatic --germline --tools delly`; to use MuTect2, Manta, and Strelka2, do `--somatic --tools mutect2,manta,strelka2`. ## Components From 07a6c3c1fe7b832c498c7debcb64a45aa91bbdc5 Mon Sep 17 00:00:00 2001 From: Ivkovic Date: Thu, 23 May 2019 18:00:31 -0400 Subject: [PATCH 3/6] fix delly container --- containers/delly/Dockerfile | 70 +++++++++++++++++++++++-------------- 1 file changed, 43 insertions(+), 27 deletions(-) diff --git a/containers/delly/Dockerfile b/containers/delly/Dockerfile index 048613538..bd58415c6 100644 --- a/containers/delly/Dockerfile +++ b/containers/delly/Dockerfile @@ -1,32 +1,48 @@ -FROM frolvlad/alpine-glibc:alpine-3.9 +FROM ubuntu:18.04 -LABEL maintainer="Nikhil Kumar (kumarn1@mskcc.org)" \ - version.image="1.0.0" \ - version.delly="0.7.7" \ - version.htslib="1.6" \ - version.bcftools="1.6" \ - version.samtools="1.6" \ - version.alpine="3.8" \ - source.delly="https://github.com/dellytools/delly/releases/tag/v0.7.7" \ - source.htslib="https://github.com/samtools/htslib/releases/tag/1.6" \ - source.bcftools="https://github.com/samtools/bcftools/releases/tag/1.6" \ - source.samtools="https://github.com/samtools/samtools/releases/tag/1.6" +LABEL maintainer="Sinisa Ivkovic (ivkovics@mskcc.org)" \ + version.ubuntu="18.04" \ + version.delly="0.8.1" \ + version.htslib="1.9" -ENV DELLY_VERSION v0.8.1 +# set the environment variables +ENV delly_version 0.8.1 +ENV htslib_version 1.9 +# run update +RUN apt-get update -y && apt-get install -y \ + libnss-sss \ + curl \ + less \ + vim \ + wget \ + unzip \ + build-essential \ + libboost-all-dev \ + zlib1g-dev \ + libncurses5-dev \ + libncursesw5-dev \ + libnss-sss \ + libbz2-dev \ + liblzma-dev \ + bzip2 \ + libcurl4-openssl-dev -# set environment -ENV BOOST_ROOT /usr -ENV SEQTK_ROOT /opt/htslib - -RUN apk add --update --no-cache ncurses \ - && apk add --virtual=deps --update --no-cache ncurses-dev musl-dev git g++ make zlib-dev \ - && apk add bash build-base asciidoc bzip2-dev xz-dev boost-dev \ - && cd /opt \ - # install delly - && cd /opt \ - && git clone --recursive https://github.com/dellytools/delly.git -b ${DELLY_VERSION} \ - && cd /opt/delly/ \ - && make all \ - && install -p /opt/delly/src/delly /usr/local/bin/ +# install htslib +WORKDIR /usr/local/bin/ +RUN curl -SL https://github.com/samtools/htslib/releases/download/${htslib_version}/htslib-${htslib_version}.tar.bz2 \ + > /usr/local/bin/htslib-${htslib_version}.tar.bz2 +RUN tar -xjf /usr/local/bin/htslib-${htslib_version}.tar.bz2 -C /usr/local/bin/ +RUN cd /usr/local/bin/htslib-${htslib_version}/ && ./configure +RUN cd /usr/local/bin/htslib-${htslib_version}/ && make +RUN cd /usr/local/bin/htslib-${htslib_version}/ && make install +ENV LD_LIBRARY_PATH /usr/local/bin/htslib-${htslib_version}/ +# install delly +WORKDIR /usr/local/bin +RUN wget https://github.com/dellytools/delly/archive/v${delly_version}.zip +RUN unzip v${delly_version}.zip +WORKDIR /usr/local/bin/delly-${delly_version} +RUN make all +WORKDIR /usr/local/bin +RUN ln -s /usr/local/bin/delly-0.8.1/src/delly /usr/local/bin/delly From 18f189eb0786b766610ec2e873d4a226b051205b Mon Sep 17 00:00:00 2001 From: evanbiederstedt Date: Fri, 24 May 2019 15:42:09 -0400 Subject: [PATCH 4/6] updated Dockerfile for facets-suite, using v 1.5.7 --- containers/facets-suite/Dockerfile | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/containers/facets-suite/Dockerfile b/containers/facets-suite/Dockerfile index 86c3a5de5..7f9e87c42 100644 --- a/containers/facets-suite/Dockerfile +++ b/containers/facets-suite/Dockerfile @@ -2,16 +2,16 @@ FROM frolvlad/alpine-glibc:alpine-3.9 LABEL maintainer="Nikhil Kumar (kumarn1@mskcc.org)" \ version.image="1.0.0" \ - version.facets_suite="1.5.6" \ + version.facets_suite="1.5.7" \ version.facets="0.5.14"\ version.alpine="3.8" \ version.r="3.5.1" \ version.pctGCdata="0.2.0" \ - source.facets_suite="https://github.com/mskcc/facets-suite/releases/tag/1.5.6" \ + source.facets_suite="https://github.com/mskcc/facets-suite/releases/tag/1.5.7" \ source.facets="https://github.com/mskcc/facets/archive/v0.5.14.tar.gz"\ source.r="https://pkgs.alpinelinux.org/package/edge/community/x86/R" -ENV FACETS_SUITE_VERSION 1.5.6 +ENV FACETS_SUITE_VERSION 1.5.7 ENV FACETS_VERSION 0.5.14 ENV PCTGCDATA 0.2.0 From ed0a783536ba4c6845fd4c119da64796290c2bdf Mon Sep 17 00:00:00 2001 From: evanbiederstedt Date: Fri, 24 May 2019 16:57:28 -0400 Subject: [PATCH 5/6] revised, use "cp -r /tmp/facets-suite-${FACETS_SUITE_VERSION}/*" --- containers/facets-suite/Dockerfile | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/containers/facets-suite/Dockerfile b/containers/facets-suite/Dockerfile index 7f9e87c42..b3dc8e16e 100644 --- a/containers/facets-suite/Dockerfile +++ b/containers/facets-suite/Dockerfile @@ -24,6 +24,8 @@ RUN apk add --update \ && apk add cairo cairo-dev libxt-dev libxml2-dev font-xfree86-type1 \ # download and install R && apk add R R-dev \ + # change permissions for /tmp; still doesn't resolve R warnings + && chmod 777 -R /tmp \ # download and unzip facets, facets-suite, pctGCdata && cd /tmp \ && wget https://github.com/mskcc/facets-suite/archive/${FACETS_SUITE_VERSION}.tar.gz -O facets-suite-${FACETS_SUITE_VERSION}.tar.gz \ @@ -53,7 +55,7 @@ RUN apk add --update \ && sed -i "s/opt\/common\/CentOS_6-dev\/R\/R-3.2.2\//usr\//g" *.R \ # copy execs to /usr/bin/facets-suite && mkdir -p /usr/bin/facets-suite/ \ - && cp /tmp/facets-suite-${FACETS_SUITE_VERSION}/* /usr/bin/facets-suite/ \ + && cp -r /tmp/facets-suite-${FACETS_SUITE_VERSION}/* /usr/bin/facets-suite/ \ # clean up && rm -rf /var/cache/apk/* /tmp/* From a20701c596234c0a3a73802846e4291bfddb0c10 Mon Sep 17 00:00:00 2001 From: Allan Bolipata Date: Tue, 28 May 2019 11:31:18 -0400 Subject: [PATCH 6/6] Version bump 0.13.0 --- nextflow.config | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/nextflow.config b/nextflow.config index bdee109a0..53ac7d069 100755 --- a/nextflow.config +++ b/nextflow.config @@ -18,7 +18,7 @@ manifest { mainScript = 'main_align_markDups_BaseRecal.nf' name = 'Vaporware' nextflowVersion = '>=0.32.0' - version = '0.12.0' + version = '0.13.0' } // Provides extra options for the Java and Nextflow runtime.