Feature hic scaffolding (#123)

* Add Pixi dev env

* Update/install YAHS and BWAMEM2 modules

* Add Scaffold skeleton

* Add scaffold subworkflow to main.nf

* add pairtools and pairix modules for HiC pipeline

* add more modules for scaffolding workflow

* add .DS_Store to .gitignore

* add pairtools to scaffolding subworkflow

* add initial yahs scaffolding subworkflow

* fix file names and process argument

* add version file collection

* remove hard-coded ncbi.taxdb path from test profile

* fix typo ch_versions

* increase all cpu and memory default values for scaffolding subworkflow

* increase time limits for scaffolding processes

* add publishDir for yahs process

* fix stage name for yahs publishDir

* publish hic alignment statistics and hard code the threads for compression and decomression tasks

* bump up pairtools version to  1.1.0

* add tiny read set: PacBio HiFi and HiC, species: drosophila, 2Mb

* Add testing (#126)

* Initialise nf-test

* Remove defunct tests

* Add test data provenance README

* Update docker settings

* Docker as default profile

* Add Tiny CI test yaml

* Update test config

* Update pairtools patch

* Update gitignore

* Comment out cpus

* Add Resume to test config

* Fix intermittent null object issue with LazyMap

* Move around resource configurations

* Improve process tags

* Clean up resource allocation

* Add config README

* Fix pairtools parse cpu allocation

* Update tests/data/tiny/README.md

---------

Co-authored-by: Martin Pippel <[email protected]>

* Fix Scaffolding to work in Multiple HiC

* Assembly report (#127)

* Update pairtools patch

* Move and clean up resource configurations

* Sketch out prototype

* Move Groovy things inside workflow

* Add mamba for conda development when container doesn't exist

* Add debug profile

* Add GenomeScope2 plots

* Update assets/notebooks/assembly_report.qmd

Co-authored-by: Martin Pippel <[email protected]>

* Add Pixi tasks to toml

* Add container to Quarto notebook

* Update report qmd

* Add dev task

* Add pandas to quarto env

* Change debug flag name to diagnostics

* Update report

* Transpose DToL table

* Add more tables and plots

* Add KatComp

* Add HiFiasm Kmer graph

* Add Quast

* Add Merqury{,FK} figures and tables

* Add Purgedups plot

* Add Pairtools plots

---------

Co-authored-by: Martin Pippel <[email protected]>

* Remove custom purging cutoff

* copy scaffolding subworkflow to new curation subworkflow

* add initial curation subworkflow

* cleanup view calls and add tool versions of curation subworkflow

* bugfix renamed process name aliases: SAMTOOLS_MERGE_HIC and SAMTOOLS_MERGE_HIFI

* scaffolding: add dedup stats file to output directory

* write final higlass and pretext results to output dir

* switch from combine to join channels

* cleanup code

* add default runtime limits: cpus,memory,time for some processes

* cite sanger and arima

* Update configs/modules.config

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* Update configs/modules.config

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* remove hardcoded ncbi.taxdb path

* add curly brackets

* add resourceLimits to test.config and cleanup

* Update modules/local/hic_curation/bam2bed_sort.nf

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* Update modules/local/hic_curation/bam2coverageTracks.nf

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* cite Sanger and DAMARVEL repo for BAM2BED_SORT module

* replace container = <CONT> with container <CONT>

* Update modules/local/hic_curation/create_telomer_hitile_track.nf

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* Update modules/local/hic_curation/pretext_tracks_ingestion.nf

Co-authored-by: Mahesh Binzer-Panchal <[email protected]>

* add comment to transpose(by:1)

* add comments about joining

* merge conflict

* bugfix add missing mapQV value two two_read_combiner

* bugfix - groupTuples on bam files

* Add test resource limits (#132)

* Update test profile limits

* Add resource limits to Gitpod

* small bugfixed

* bugfix convert string to integer

* increase memory for FASTK

* bugfix FASTK resources: issue compressed reads are written out uncompressed to disk

* reduce cpus for meryl count process

* reduce memory limit for bwa index process

* fix read variable and reduce memory in meryl count module

* fix MemoryUnit multiplication

* fix MemoryUnit multiplication - again

* create bigwig files instead of hitile file - solve issue #133

* cleanup

* add Maheshs Quarto fix from #aa25974

* add exception for FCSGX resource to all profiles

* bugfix: add commas to configs

* bugfix missing closing bracket

* another bugfix trial

* increase memory limit for MINIMAP2_ALIGN_ASSEMBLY_PRIMARY

* bugfix FCSGX_RUNGX runtime limits

* skip hitile creation, use bigwig insteadf

* cleanup uppmax.config, force FCSGX_RUNGX jobs to run on node partition on rackham

* Follow dardel config update from nf-core/configs#803

* Move task.cpus from config to process

* Remove extra resourceLimits

* Move params.hifi_coverage_cap to process input

* Move params.hic_map_sort_by to process input:

* Fix report

* Fix decontaminate subworkflow

* cleanup, remove -5 bwa default argument, remove samtools fixmate step, add ugly grep hack to avoid samtools header issue

* bugfix args3

---------

Co-authored-by: MartinPippel <[email protected]>
Co-authored-by: Martin Pippel <[email protected]>

.gitattributes

@@ -1 +1,3 @@
+# GitHub syntax highlighting
+pixi.lock linguist-language=YAML linguist-generated=true
 *.config linguist-language=nextflow

.gitignore

@@ -1,3 +1,19 @@
+# Nextflow files
 .nextflow*
 work/
-results/
+
+# Pipeline files
+results/
+*_cache/
+
+# Pixi environments
+.pixi/
+*.egg-info
+
+# nf-test files
+.nf-test.log
+.nf-test/
+
+# misc
+.DS_Store
+.screenrc