[New Workflow] Adding the TheiaMeta_Panel_Illumina_PE Workflow

.dockstore.yml

@@ -230,6 +230,11 @@ workflows:
    primaryDescriptorPath: /workflows/theiameta/wf_theiameta_illumina_pe.wdl
    testParameterFiles:
     - /tests/inputs/empty.json
+ - name: TheiaMeta_Panel_Illumina_PE_PHB
+   subclass: WDL
+   primaryDescriptorPath: /workflows/theiameta/wf_theiameta_panel_illumina_pe.wdl
+   testParameterFiles:
+    - /tests/inputs/empty.json
  - name: Snippy_Streamline_PHB
    subclass: WDL
    primaryDescriptorPath: /workflows/phylogenetics/wf_snippy_streamline.wdl

docs/workflows/genomic_characterization/theiameta.md

@@ -247,7 +247,6 @@ The TheiaMeta_Illumina_PE workflow processes Illumina paired-end (PE) reads ge
     `metaspades` is a _de novo_ assembler that first constructs a de Bruijn graph of all the reads using the SPAdes algorithm. Through various graph simplification procedures, paths in the assembly graph are reconstructed that correspond to long genomic fragments within the metagenome. For more details, please see the original publication.
 
     !!! techdetails "MetaSPAdes Technical Details"
-
         |  | Links |
         | --- | --- |
         | Task | [task_metaspades.wdl](https://github.com/theiagen/public_health_bioinformatics/blob/main/tasks/assembly/task_metaspades.wdl) |

docs/workflows_overview/workflows_alphabetically.md

@@ -43,6 +43,7 @@ title: Alphabetical Workflows
 | [**TheiaCov Workflow Series**](../workflows/genomic_characterization/theiacov.md) | Viral genome assembly, QC and characterization from amplicon sequencing | HIV, Influenza, Monkeypox virus, RSV-A, RSV-B, SARS-CoV-2, Viral, WNV | Sample-level, Set-level | Some optional features incompatible, Yes | v2.2.0 | [TheiaCoV_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_PE_PHB:main?tab=info), [TheiaCoV_Illumina_SE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_SE_PHB:main?tab=info), [TheiaCoV_ONT_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ONT_PHB:main?tab=info), [TheiaCoV_ClearLabs_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ClearLabs_PHB:main?tab=info), [TheiaCoV_FASTA_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_PHB:main?tab=info), [TheiaCoV_FASTA_Batch_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_Batch_PHB:main?tab=info) |
 | [**TheiaEuk**](../workflows/genomic_characterization/theiaeuk.md) | Mycotic genome assembly, QC and characterization from WGS data | Mycotics | Sample-level | Some optional features incompatible, Yes | v2.0.1 | [TheiaEuk_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaEuk_Illumina_PE_PHB:main?tab=info) |
 | [**TheiaMeta**](../workflows/genomic_characterization/theiameta.md) | Genome assembly and QC from metagenomic sequencing | Any taxa | Sample-level | Yes | v2.0.0 | [TheiaMeta_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaMeta_Illumina_PE_PHB:main?tab=info) |
+| [**TheiaMeta Panel**](../workflows/genomic_characterization/theiameta_panel.md) | Genome assembly and QC from metagenomic sequencing using a panel | Viral | Sample-level | Yes |2.X.X | [TheiaMeta_Panel_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaMeta_Panel_PHB:main?tab=info) |
 | [**TheiaProk Workflow Series**](../workflows/genomic_characterization/theiaprok.md) | Bacterial genome assembly, QC and characterization from WGS data | Bacteria | Sample-level | Some optional features incompatible, Yes | v2.2.0 | [TheiaProk_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_Illumina_PE_PHB:main?tab=info), [TheiaProk_Illumina_SE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_Illumina_SE_PHB:main?tab=info), [TheiaProk_ONT_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_ONT_PHB:main?tab=info), [TheiaProk_FASTA_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_FASTA_PHB:main?tab=info) |
 | [**TheiaValidate**](../workflows/standalone/theiavalidate.md)| This workflow performs basic comparisons between user-designated columns in two separate tables. | Any taxa | | No | v2.0.0 | [TheiaValidate_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaValidate_PHB:main?tab=info) |
 | [**Transfer_Column_Content**](../workflows/data_export/transfer_column_content.md)| Transfer contents of a specified Terra data table column for many samples ("entities") to a GCP storage bucket location | Any taxa | Set-level | Yes | v1.3.0 | [Transfer_Column_Content_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/Transfer_Column_Content_PHB:main?tab=info) |

docs/workflows_overview/workflows_kingdom.md

@@ -88,6 +88,7 @@ title: Workflows by Kingdom
 | [**Terra_2_GISAID**](../workflows/public_data_sharing/terra_2_gisaid.md)| Upload of assembly data to GISAID | SARS-CoV-2, Viral | Set-level | Yes | v1.2.1 | [Terra_2_GISAID_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/Terra_2_GISAID_PHB:main?tab=info) |
 | [**Terra_2_NCBI**](../workflows/public_data_sharing/terra_2_ncbi.md)| Upload of sequence data to NCBI | Bacteria, Mycotics, Viral | Set-level | No | v2.1.0 | [Terra_2_NCBI_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/Terra_2_NCBI_PHB:main?tab=info) |
 | [**TheiaCov Workflow Series**](../workflows/genomic_characterization/theiacov.md) | Viral genome assembly, QC and characterization from amplicon sequencing | HIV, Influenza, Monkeypox virus, RSV-A, RSV-B, SARS-CoV-2, Viral, WNV | Sample-level, Set-level | Some optional features incompatible, Yes | v2.2.0 | [TheiaCoV_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_PE_PHB:main?tab=info), [TheiaCoV_Illumina_SE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_SE_PHB:main?tab=info), [TheiaCoV_ONT_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ONT_PHB:main?tab=info), [TheiaCoV_ClearLabs_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ClearLabs_PHB:main?tab=info), [TheiaCoV_FASTA_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_PHB:main?tab=info), [TheiaCoV_FASTA_Batch_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_Batch_PHB:main?tab=info) |
+| [**TheiaMeta Panel**](../workflows/genomic_characterization/theiameta_panel.md) | Genome assembly and QC from metagenomic sequencing using a panel | Viral | Sample-level | Yes | v2.X.X | [TheiaMeta_Panel_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaMeta_Panel_PHB:main?tab=info) |
 | [**Usher_PHB**](../workflows/phylogenetic_placement/usher.md)| Use UShER to rapidly and accurately place your samples on any existing phylogenetic tree | Monkeypox virus, SARS-CoV-2, Viral | Sample-level, Set-level | Yes | v2.1.0 | [Usher_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/Usher_PHB:main?tab=info) |
 | [**VADR_Update**](../workflows/genomic_characterization/vadr_update.md)| Update VADR assignments | HAV, Influenza, Monkeypox virus, RSV-A, RSV-B, SARS-CoV-2, Viral, WNV | Sample-level | Yes | v1.2.1 | [VADR_Update_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/VADR_Update_PHB:main?tab=info) |
 

docs/workflows_overview/workflows_type.md

@@ -30,6 +30,7 @@ title: Workflows by Type
 | [**TheiaCov Workflow Series**](../workflows/genomic_characterization/theiacov.md) | Viral genome assembly, QC and characterization from amplicon sequencing | HIV, Influenza, Monkeypox virus, RSV-A, RSV-B, SARS-CoV-2, Viral, WNV | Sample-level, Set-level | Some optional features incompatible, Yes | v2.2.0 | [TheiaCoV_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_PE_PHB:main?tab=info), [TheiaCoV_Illumina_SE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_Illumina_SE_PHB:main?tab=info), [TheiaCoV_ONT_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ONT_PHB:main?tab=info), [TheiaCoV_ClearLabs_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_ClearLabs_PHB:main?tab=info), [TheiaCoV_FASTA_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_PHB:main?tab=info), [TheiaCoV_FASTA_Batch_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaCoV_FASTA_Batch_PHB:main?tab=info) |
 | [**TheiaEuk**](../workflows/genomic_characterization/theiaeuk.md) | Mycotic genome assembly, QC and characterization from WGS data | Mycotics | Sample-level | Some optional features incompatible, Yes | v2.0.1 | [TheiaEuk_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaEuk_Illumina_PE_PHB:main?tab=info) |
 | [**TheiaMeta**](../workflows/genomic_characterization/theiameta.md) | Genome assembly and QC from metagenomic sequencing | Any taxa | Sample-level | Yes | v2.0.0 | [TheiaMeta_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaMeta_Illumina_PE_PHB:main?tab=info) |
+| [**TheiaMeta Panel**](../workflows/genomic_characterization/theiameta_panel.md) | Genome assembly and QC from metagenomic sequencing using a panel | Viral | Sample-level | Yes | v2.X.X | [TheiaMeta_Panel_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaMeta_Panel_PHB:main?tab=info) |
 | [**TheiaProk Workflow Series**](../workflows/genomic_characterization/theiaprok.md) | Bacterial genome assembly, QC and characterization from WGS data | Bacteria | Sample-level | Some optional features incompatible, Yes | v2.2.0 | [TheiaProk_Illumina_PE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_Illumina_PE_PHB:main?tab=info), [TheiaProk_Illumina_SE_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_Illumina_SE_PHB:main?tab=info), [TheiaProk_ONT_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_ONT_PHB:main?tab=info), [TheiaProk_FASTA_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/TheiaProk_FASTA_PHB:main?tab=info) |
 | [**VADR_Update**](../workflows/genomic_characterization/vadr_update.md)| Update VADR assignments | HAV, Influenza, Monkeypox virus, RSV-A, RSV-B, SARS-CoV-2, Viral, WNV | Sample-level | Yes | v1.2.1 | [VADR_Update_PHB](https://dockstore.org/workflows/github.com/theiagen/public_health_bioinformatics/VADR_Update_PHB:main?tab=info) |
 

mkdocs.yml

@@ -25,6 +25,7 @@ nav:
                - TheiaCoV Workflow Series: workflows/genomic_characterization/theiacov.md
                - TheiaEuk Workflow Series: workflows/genomic_characterization/theiaeuk.md
                - TheiaMeta: workflows/genomic_characterization/theiameta.md
+               - TheiaMeta_Panel: workflows/genomic_characterization/theiameta_panel.md
                - TheiaProk Workflow Series: workflows/genomic_characterization/theiaprok.md
                - VADR_Update: workflows/genomic_characterization/vadr_update.md
             - Phylogenetic Construction:
@@ -115,6 +116,7 @@ nav:
                - Terra_2_GISAID: workflows/public_data_sharing/terra_2_gisaid.md
                - Terra_2_NCBI: workflows/public_data_sharing/terra_2_ncbi.md
                - TheiaCoV Workflow Series: workflows/genomic_characterization/theiacov.md
+               - TheiaMeta_Panel: workflows/genomic_characterization/theiameta_panel.md
                - Usher_PHB: workflows/phylogenetic_placement/usher.md
                - VADR_Update: workflows/genomic_characterization/vadr_update.md
         - Workflows Alphabetically: 
@@ -152,6 +154,7 @@ nav:
             - TheiaCoV Workflow Series: workflows/genomic_characterization/theiacov.md
             - TheiaEuk Workflow Series: workflows/genomic_characterization/theiaeuk.md
             - TheiaMeta: workflows/genomic_characterization/theiameta.md
+            - TheiaMeta_Panel: workflows/genomic_characterization/theiameta_panel.md
             - TheiaProk Workflow Series: workflows/genomic_characterization/theiaprok.md
             - TheiaValidate: workflows/standalone/theiavalidate.md
             - Transfer_Column_Content: workflows/data_export/transfer_column_content.md

tasks/assembly/task_metaspades.wdl

@@ -15,23 +15,35 @@ task metaspades_pe {
   }
   command <<<
     metaspades.py --version | head -1 | cut -d ' ' -f 2 | tee VERSION
-    metaspades.py \
+    touch WARNING
+
+    if metaspades.py \
       -1 ~{read1_cleaned} \
       -2 ~{read2_cleaned} \
       ~{'-k ' + kmers} \
       -m ~{memory} \
       -t ~{cpu} \
       -o metaspades \
       --phred-offset ~{phred_offset} \
-      ~{metaspades_opts}
+      ~{metaspades_opts}; then
+
+      mv metaspades/contigs.fasta ~{samplename}_contigs.fasta
 
-    mv metaspades/contigs.fasta ~{samplename}_contigs.fasta
+      if [ ! -s ~{samplename}_contigs.fasta ]; then
+        tee "Metaspades produced an empty assembly for ~{samplename}" > WARNING
+        rm -f ~{samplename}_contigs.fasta
+      fi
+
+    else
+      tee "Metaspades failed to assemble for ~{samplename}" > WARNING
+    fi
 
   >>>
   output {
-    File assembly_fasta = "~{samplename}_contigs.fasta"
+    File? assembly_fasta = "~{samplename}_contigs.fasta"
     String metaspades_version = read_string("VERSION")
     String metaspades_docker = '~{docker}'
+    String metaspades_warning = read_string("WARNING")
   }
   runtime {
     docker: "~{docker}"

tasks/quality_control/basic_statistics/task_fastq_scan.wdl

@@ -4,13 +4,14 @@ task fastq_scan_pe {
   input {
     File read1
     File read2
-    String read1_name = basename(basename(basename(read1, ".gz"), ".fastq"), ".fq")
-    String read2_name = basename(basename(basename(read2, ".gz"), ".fastq"), ".fq")
-    Int disk_size = 50
+
+    Int disk_size = 100
     String docker = "us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan:1.0.1--h4ac6f70_3"
     Int memory = 2
     Int cpu = 1
   }
+  String read1_name = basename(basename(basename(read1, ".gz"), ".fastq"), ".fq")
+  String read2_name = basename(basename(basename(read2, ".gz"), ".fastq"), ".fq")
   command <<<
     # exit task in case anything fails in one-liners or variables are unset
     set -euo pipefail
@@ -77,12 +78,13 @@ task fastq_scan_pe {
 task fastq_scan_se {
   input {
     File read1
-    String read1_name = basename(basename(basename(read1, ".gz"), ".fastq"), ".fq")
-    Int disk_size = 50
+
+    Int disk_size = 100
     Int memory = 2
     Int cpu = 1
     String docker = "us-docker.pkg.dev/general-theiagen/biocontainers/fastq-scan:1.0.1--h4ac6f70_3"
   }
+  String read1_name = basename(basename(basename(read1, ".gz"), ".fastq"), ".fq")
   command <<<
     # exit task in case anything fails in one-liners or variables are unset
     set -euo pipefail

tasks/quality_control/read_filtering/task_pilon.wdl

@@ -16,20 +16,26 @@ task pilon {
     pilon --version | cut -d' ' -f3 | tee VERSION
 
     # run pilon
-    pilon \
+    if pilon \
     --genome ~{assembly} \
     --frags ~{bam} \
     --output ~{samplename} \
     --outdir pilon \
-    --changes --vcf
+    --changes --vcf; then
+      touch WARNING
+    else
+      tee "Pilon failed to run for ~{samplename}" > WARNING
+      exit 1
+    fi
 
   >>>
   output {
-    File assembly_fasta = "pilon/~{samplename}.fasta"
-    File changes = "pilon/~{samplename}.changes"
-    File vcf = "pilon/~{samplename}.vcf"
+    File? assembly_fasta = "pilon/~{samplename}.fasta"
+    File? changes = "pilon/~{samplename}.changes"
+    File? vcf = "pilon/~{samplename}.vcf"
     String pilon_version = read_string("VERSION")
     String pilon_docker = "~{docker}"
+    String pilon_warning = read_string("WARNING")
   }
   runtime {
     docker: "~{docker}"

tasks/species_typing/betacoronavirus/task_pangolin.wdl

@@ -8,9 +8,9 @@ task pangolin4 {
     Float max_ambig = 0.5
     String docker
     String? analysis_mode
-    Boolean expanded_lineage=true
-    Boolean skip_scorpio=false
-    Boolean skip_designation_cache=false
+    Boolean expanded_lineage = true
+    Boolean skip_scorpio = false
+    Boolean skip_designation_cache = false
     String? pangolin_arguments
     Int disk_size = 100
     Int memory = 8

tasks/taxon_id/contamination/task_kraken2.wdl

@@ -168,8 +168,8 @@ task kraken2_standalone {
     File kraken2_unclassified_read1 = "~{samplename}.unclassified_1.fastq.gz"
     File? kraken2_unclassified_read2 = "~{samplename}.unclassified_2.fastq.gz"
     File kraken2_classified_read1 = "~{samplename}.classified_1.fastq.gz"
-    Float kraken2_percent_human = read_float("PERCENT_HUMAN")
     File? kraken2_classified_read2 = "~{samplename}.classified_2.fastq.gz"
+    Float kraken2_percent_human = read_float("PERCENT_HUMAN")
     String kraken2_database = kraken2_db
   }
   runtime {

tasks/taxon_id/task_krakentools.wdl

@@ -0,0 +1,61 @@
+version 1.0
+
+task extract_kraken_reads {
+  input {
+    File kraken2_output
+    File kraken2_report
+    File read1
+    File read2
+    Int taxon_id
+
+    Int cpu = 1
+    Int disk_size = 100
+    String docker = "us-docker.pkg.dev/general-theiagen/theiagen/krakentools:d4a2fbe"
+    Int memory = 4
+  }
+  command <<<
+    gunzip -c ~{kraken2_output} > kraken2_output_unzipped.txt
+
+    python3 /KrakenTools/extract_kraken_reads.py \
+      -k kraken2_output_unzipped.txt \
+      -s1 ~{read1} \
+      -s2 ~{read2} \
+      --taxid ~{taxon_id} \
+      --report ~{kraken2_report} \
+      --include-parents \
+      --include-children \
+      --fastq-output \
+      --output ~{taxon_id}_1.fastq \
+      --output2 ~{taxon_id}_2.fastq
+
+    if [ -s ~{taxon_id}_1.fastq ]; then
+      echo "DEBUG: Taxon ~{taxon_id} reads extracted"
+      echo "true" > CONTINUE
+
+      gzip ~{taxon_id}_1.fastq 
+      gzip ~{taxon_id}_2.fastq
+    else
+      echo "DEBUG: No reads were extracted for taxon ~{taxon_id}, removing empty files"
+      echo "false" > CONTINUE
+    fi
+
+    grep ~{taxon_id} ~{kraken2_report} | awk '{for (i=6; i <= NF; ++i) print $i}' | tr '\n' ' ' | xargs > ORGANISM_NAME
+
+  >>>
+  output {
+    File? extracted_read1 = "~{taxon_id}_1.fastq.gz"
+    File? extracted_read2 = "~{taxon_id}_2.fastq.gz"
+    String organism_name = read_string("ORGANISM_NAME")
+    String krakentools_docker = docker
+    Boolean success = read_boolean("CONTINUE")
+  }
+  runtime {
+    cpu: cpu
+    disk: disk_size + " GB"
+    disks: "local-disk " + disk_size + " SSD"
+    docker: docker
+    memory: "~{memory} GB"
+    preemptible: 1
+    maxRetries: 3
+  }
+}