[TheiaProk] Adds stxtyper to merlin_magic and TheiaProk wfs

[kapsakcj]

## Please keep this PR as draft for now as stxtyper is still undergoing validation/peer review/publication

Using this draft PR for tracking stxtyper development as well as amrfinderplus (that runs stxtyper under the hood). NOTE: ignore the amrfinderplus mention, that update is NOT included in this PR

Our partners are actively using this branch for testing purposes. Let's keep it available for now

I'll try to keep this branch up-to-date with main to incorporate other changes & resolve merge conflicts if they arise.

This PR closes #443

🗑️ This dev branch should NOT be deleted after merging to main.

2024-09-23 update: I expect to hear feedback from our partner soon, but updating PR message now with tests and info

🧠 Aim, Context and Functionality

This PR adds Stxtyper to the TheiaProk workflows. Stxtyper is used to detect and type shiga toxin genes in bacterial genome assemblies. It also attempts to detect novel shiga toxin subtypes in cases where the detected sequences diverge from the reference sequences.

These genes are usually found in E. coli (STEC), but can also be found in Shigella species as well as some other genera more rarely, like Klebsiella. It is developed by NCBI in collaboration with a number of different groups including CDC, FDA, SSI, and others. A publication to fully describe the tool and it's validation is in the works but a software release has been made so the community may test the software further and begin using the tool.

This tool queries genome assemblies for 2 genes or subunits involved in shiga toxin production, stxA and stxB. The A subunit is longer than the B subunit. Stxtyper attempts to detect these, compare them to a database of known sequences, and type them based on amino acid composition. The typing algorithm will be described in the publication when it is published.

More info & source code found here: https://github.com/ncbi/stxtyper

To learn more about shiga toxin subtypes and the description of the latest subtypes, Stx2n, Stx2j, Stx2m, and Stx2o, see this publication (shamless plug): https://www.mdpi.com/2076-2607/11/10/2561

Eventually this tool will be incorporated into AMRFinderPlus and will run behind-the-scenes when the user provides the amrfinder --organism Escherichia option, but we wanted the functionality now and the ability to run separate from AMRFinderPlus.

🛠️ Impacted Workflows/Tasks & Changes Being Made

This will affect the behavior of the workflow(s) even if users don’t change any workflow inputs relative to the last version : Yes/No

Running this workflow on different occasions could result in different results, e.g. due to use of a live database, "latest" docker image, or stochastic data processing : Yes/No

📋 Workflow/Task Step Changes

🔄 Data Processing

New task. StxTyper is run on assemblies and output TSV is parsed to generate a few string output columns

Docker/software or software versions changed: New

Databases or database versions changed: New

Data processing/commands changed: New

File processing changed: New

Compute resources changed: 1cpu, 4GB RAM. Doesn't need much. Also enabled preemptible since it usually runs in less than 3 minutes

➡️ Inputs

new TheiaProk wf inputs:

Terra Task name	Variable	Type	Description	Default value	Terra Status	Workflow
merlin_magic	call_shigeifinder_reads_input	Boolean	If set to "true", the ShigEiFinder task will run again but using read files as input instead of the assembly file. Input is shown but not used for TheiaProk_FASTA.	FALSE	Optional	FASTA, ONT, PE, SE
merlin_magic	call_stxtyper	Boolean	If set to "true", the StxTyper task will run on all samples regardless of the gambit_predicted_taxon output. Useful if you suspect a non-E.coli or non-Shigella sample contains stx genes.	FALSE	Optional	FASTA, ONT, PE, SE
merlin_magic	stxtyper_cpu	Int	The number of CPU cores to allocate for the task.	1	Optional	FASTA, ONT, PE, SE
merlin_magic	stxtyper_disk_size	Int	Amount of storage (in GB) to allocate to the task	50	Optional	FASTA, ONT, PE, SE
merlin_magic	stxtyper_docker_image	String	The Docker container to use for the task	us-docker.pkg.dev/general-theiagen/staphb/stxtyper:1.0.24	Optional	FASTA, ONT, PE, SE
merlin_magic	stxtyper_enable_debug	Boolean	When enabled, additional messages are printed and files in $TMPDIR are not removed after running	FALSE	Optional	FASTA, ONT, PE, SE
merlin_magic	stxtyper_memory	Int	Amount of memory (in GB) to allocate to the task	4	Optional	FASTA, ONT, PE, SE

⬅️ Outputs

New TheiaProk wf outputs:

Variable	Type	Description	Workflow
stxtyper_all_hits	String	Comma-separated list of matches of all types. Includes complete, partial, frameshift, internal stop, and novel hits. List is de-duplicated so multiple identical hits are only listed once. For example if 5 partial stx2 hits are detected in the genome, only 1 "stx2" will be listed in this field. To view the potential subtype for each partial hit, the user will need to view the stxtyper_report TSV file.	FASTA, ONT, PE, SE
stxtyper_complete_operons	String	Comma-separated list of all COMPLETE operons detected by StxTyper. Show multiple hits if present in results.	FASTA, ONT, PE, SE
stxtyper_docker	String	Name of docker image used by the stxtyper task.	FASTA, ONT, PE, SE
stxtyper_novel_hits	String	Comma-separated list of matches that have the OPERON output of "COMPLETE_NOVEL". Possible outputs "stx1", "stx2", or "stx1,stx2"	FASTA, ONT, PE, SE
stxtyper_num_hits	Int	Number of "hits" or rows present in the stxtyper_report TSV file	FASTA, ONT, PE, SE
stxtyper_partial_hits	String	Possible outputs "stx1", "stx2", or "stx1,stx2". Tells the user that there was a partial hit to either the A or B subunit, but does not describe which subunit, only the possible types from the PARTIAL matches.	FASTA, ONT, PE, SE
stxtyper_report	File	Raw results TSV file produced by StxTyper	FASTA, ONT, PE, SE
stxtyper_stx_frameshifts_or_internal_stop_hits	String	Comma-separated list of matches that have the OPERON output of "FRAMESHIFT" or "INTERNAL_STOP". Possible outputs "stx1", "stx2", or "stx1,stx2"	FASTA, ONT, PE, SE
stxtyper_version	String	Version of StxTyper used	FASTA, ONT, PE, SE

🧪 Testing

Test Dataset

A large Illumina PE dataset of 161 samples known to be pcr-positive for stx were used to test. There are a few with 0 hits and these are likely due to false PCR positives, or where the sequence data was not good enough to assemble the stx gene regions of the genome.

ONT data was from random Shigella samples from SRA, so no expectations to have stx genes in those

Commandline Testing with MiniWDL or Cromwell (optional)

Tested fine locally with miniwdl, not including output/results here in PR message. Lots of Terra testing was done.

Terra Testing

• ✅ ILMN PE with E. coli samples known to be stx positive: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-Aug2024/job_history/c5553aa7-3a9c-4a1b-b6b3-df2882f89d77
• ✅ ONT on some random Shigella that are likely stx negative https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-Aug2024/job_history/a97ec720-6d2c-4a81-9f7c-8870bb5220a5
• ✅ FASTA with same E. coli dataset that are stx positive: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-Aug2024/job_history/683989af-6753-49c3-ab9a-d0dc9403261b
• ✅ ILMN SE: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-Aug2024/job_history/9ea4668c-e670-4162-8d81-9d9d081d4297
  • Need to review wf outputs, but should be OK.
  • workflow ran successfully and stxtyper outputs look as expected

Suggested Scenarios for Reviewer to Test

Test any of the TheiaProk workflows. E. coli and Shigella spp samples would be ideal, but you can run other bacterial taxon through and set call_stxtyper optional input to true to test functionality.

Here's ~5500+ samples on NCBI pathogen detection browser that contain a hit to stx in their genome, and would be good to test with: https://www.ncbi.nlm.nih.gov/pathogens/isolates/#virulence_genotypes:stx* There's even some S. aureus and Salmonella that would be interesting to test with

Theiagen Version Release Testing (optional)

• Will changes require functional or validation testing (checking outputs etc) during the release? Yes, it would be good to add a couple STECs to our validation testing dataset. These data are public, so we could add a few to the theiaprok illumina pe validation dataset
• Do new samples need to be added to validation datasets? If so, upload these to the appropriate validation workspace Google bucket (). Please describe the new samples here and why these have been chosen.
  • 2 stx positive E. coli samples added to 2 data tables theiaprok_illumina_pe and theiaprok_illumina_se in this PHB validation template Terra workspace: https://app.terra.bio/#workspaces/theiagen-validations/PHB_Validation_v2_TEMPLATE/data . I also copied the FASTQs to our public bucket. 2 new sample names are: SRR16579222_Ecoli_stxPos2completeOperons and SRR28017575_Ecoli_stxPos1completeOperon
• Are there any output files that should be checked after running the version release testing?

🔬 Final Developer Checklist

• The workflow/task has been tested locally and results, including file contents, are as anticipated
• The workflow/task has been tested on Terra and results, including file contents, are as anticipated
• The CI/CD has been adjusted and tests are passing (to be completed by Theiagen developer)
• Code changes follow the style guide

🎯 Reviewer Checklist

• All impacted workflows/tasks have been tested on Terra with a different dataset than used for development
• All reviewer-suggested scenarios have been tested and any additional
• All changed results have been confirmed to be accurate
• All workflows/tasks impacted by change/s have been tested using a standard validation dataset to ensure no unintended change of functionality
• All code adheres to the style guide
• MD5 sums have been updated
• The PR author has addressed all comments

🗂️ Associated Documentation (to be completed by Theiagen developer)

• Relevant documentation on the Public Health Resources "PHB Main" has been updated
• Workflow diagrams have been updated to reflect changes

[kapsakcj]

other TODOs:

• first - update outputs to theiaprok ILMN PE wf
• second - update outputs to theiaprok FASTA wf
• add new outputs to theiaprok ILMN SE wf
• add new outputs to theiaprok ONT wf
• LASTLY - fix CI

[kapsakcj]

Waiting on user feedback prior to making more code changes

Plan as of 2024-07-10:

• move the call block for stxtyper to the main TheiaProk workflow WDLs, instead of calling via merlin_magic sub workflow Nevermind, planning to keep in merlin_magic subwf
• Keep stxtyper off by default, but user's can enable via boolean input call_stxtyper.

This will allow organisms of any genus/species to be screen for stx genes since they can occur in more genera/species other than E. coli & Shigella

[kapsakcj]

Also - adjust conditional in merlin_magic code to so that user can "opt-in" to running stxtyper, regardless of the taxa (i.e. gambit_predicted_taxon).

That way stxtyper is run automatically on all E. coli and Shigella and user has the ability to run it on other taxa.

[kapsakcj]

reminder to update the workflow diagram with this new tool and add it into the PR

[kapsakcj]

I expect the illumina single end tests to pass, so I'm setting this as ready for review

EDIT: they ran successfully

[kapsakcj]

gah....there are merge conflicts with the test ymls, hopefully my last commit fixes it but I may need another commit to resolve

[sage-wright]

Code changes look great -- one request: can you not add stxtyper_log to the Terra table since it's not useful to users? We can still have it as a task output so we can find it in GC but if it's not going to be used, I'm not sure if we want it to be available.

[sage-wright]

Testing on some Klebsiella here

[kapsakcj]

Code changes look great -- one request: can you not add stxtyper_log to the Terra table since it's not useful to users? We can still have it as a task output so we can find it in GC but if it's not going to be used, I'm not sure if we want it to be available.

good point, yes I'll remove it from workflow outputs

[kapsakcj]

launched another test for theiaprok_illumina_pe now that the stxtyper_log has been removed: https://app.terra.bio/#workspaces/theiagen-validations/curtis-sandbox-Aug2024/job_history/af3bef09-b239-43c3-8134-2988a2098232

I probs will have to update CI again, so commits incoming

EDIT these test workflows ran successfully and did not output the stxtyper_log file to terra data table (although it is saved in execution directory in case we ever need it)

[sage-wright]

Thanks for the quick turnaround!
⭐