Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature hic scaffolding #123

Merged
merged 85 commits into from
Nov 25, 2024
Merged

Feature hic scaffolding #123

merged 85 commits into from
Nov 25, 2024

Conversation

mahesh-panchal
Copy link
Collaborator

@mahesh-panchal mahesh-panchal commented Sep 10, 2024 •
edited
Loading

  • Adds Pixi development environment
  • Add Scaffolding Pipeline.

mahesh-panchal and others added 25 commits September 10, 2024 15:07
@mahesh-panchal
Add Pixi dev env
d2be70c
@mahesh-panchal
Update/install YAHS and BWAMEM2 modules
c5bd396
@mahesh-panchal
Add Scaffold skeleton
2163263
@mahesh-panchal
Add scaffold subworkflow to main.nf
2521f99
@MartinPippel
add pairtools and pairix modules for HiC pipeline
a680642
@MartinPippel
add more modules for scaffolding workflow
ee6395a
@MartinPippel
add .DS_Store to .gitignore
31c6321
@MartinPippel
add pairtools to scaffolding subworkflow
bb2f9b6
@MartinPippel
add initial yahs scaffolding subworkflow
f414e95
@MartinPippel
fix file names and process argument
5329c3c
@MartinPippel
add version file collection
10843e8
@MartinPippel
remove hard-coded ncbi.taxdb path from test profile
8b7daab
@MartinPippel
fix typo ch_versions
fe78d11
@MartinPippel
increase all cpu and memory default values for scaffolding subworkflow
79db0e0
@MartinPippel
increase time limits for scaffolding processes
32e7f2c
@MartinPippel
add publishDir for yahs process
38540a5
@MartinPippel
fix stage name for yahs publishDir
a053955
@MartinPippel
publish hic alignment statistics and hard code the threads for compre…
03ae6d0 
…ssion and decomression tasks
@MartinPippel
bump up pairtools version to 1.1.0
cc2a8ce
@MartinPippel
add tiny read set: PacBio HiFi and HiC, species: drosophila, 2Mb
fc2bf4a
@MartinPippel
Merge pull request #124 from MartinPippel/feature_hic_scaffolding_pai…
2236d9a 
…rtools

Feature hic scaffolding - pairtools + yahs subworkflow
@mahesh-panchal @MartinPippel
Add testing (#126)
f11fb12 
* Initialise nf-test

* Remove defunct tests

* Add test data provenance README

* Update docker settings

* Docker as default profile

* Add Tiny CI test yaml

* Update test config

* Update pairtools patch

* Update gitignore

* Comment out cpus

* Add Resume to test config

* Fix intermittent null object issue with LazyMap

* Move around resource configurations

* Improve process tags

* Clean up resource allocation

* Add config README

* Fix pairtools parse cpu allocation

* Update tests/data/tiny/README.md

---------

Co-authored-by: Martin Pippel <[email protected]>
@mahesh-panchal
Fix Scaffolding to work in Multiple HiC
b84377e
@mahesh-panchal @MartinPippel
Assembly report (#127)
2aaeeb7 
* Update pairtools patch

* Move and clean up resource configurations

* Sketch out prototype

* Move Groovy things inside workflow

* Add mamba for conda development when container doesn't exist

* Add debug profile

* Add GenomeScope2 plots

* Update assets/notebooks/assembly_report.qmd

Co-authored-by: Martin Pippel <[email protected]>

* Add Pixi tasks to toml

* Add container to Quarto notebook

* Update report qmd

* Add dev task

* Add pandas to quarto env

* Change debug flag name to diagnostics

* Update report

* Transpose DToL table

* Add more tables and plots

* Add KatComp

* Add HiFiasm Kmer graph

* Add Quast

* Add Merqury{,FK} figures and tables

* Add Purgedups plot

* Add Pairtools plots

---------

Co-authored-by: Martin Pippel <[email protected]>
@mahesh-panchal
Remove custom purging cutoff
0207444
mahesh-panchal
mahesh-panchal commented Nov 7, 2024
View reviewed changes
configs/modules.config Outdated Show resolved Hide resolved
configs/modules.config Outdated Show resolved Hide resolved
configs/modules.config Outdated Show resolved Hide resolved
configs/test.config Outdated Show resolved Hide resolved
modules/local/hic_curation/bam2coverageTracks.nf Outdated Show resolved Hide resolved
modules/local/hic_curation/create_chromosome_sizes.nf Outdated Show resolved Hide resolved
@mahesh-panchal
Copy link
Collaborator Author

Oh, and since I'm the one making the PR, you're the one who needs to approve so we can merge.

@mahesh-panchal mahesh-panchal marked this pull request as ready for review November 7, 2024 15:01
MartinPippel and others added 15 commits November 8, 2024 09:12
@mahesh-panchal
Copy link
Collaborator Author

My apologies for adding more commits, but I addressed my earlier comments, and fixed the report ( I hope ).
Hopefully you can then focus on the final fixes you want to make.

@MartinPippel
cleanup, remove -5 bwa default argument, remove samtools fixmate step…
f89c113 
…, add ugly grep hack to avoid samtools header issue
@mahesh-panchal
Copy link
Collaborator Author

Bah, Bird plots look messed up after "fixing" the notebook. Need to check again.

MartinPippel
MartinPippel approved these changes Nov 24, 2024
View reviewed changes
Copy link
Contributor

@MartinPippel MartinPippel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pipeline runs through now and creates reasonable maps. Therefore we should merge it now into the main branch.
The wasp was not a good test case as the HiC maps also look odd in the DTOL publication https://wellcomeopenresearch.org/articles/8-461
We should quickly add more features:

  • create multiple pretext-maps with different mapQV thresholds - and with/without demultiplexing
  • add more statistics about HiC quality
  • move the mapping parts (from scaffolding and map creation) into own subworkflows

@mahesh-panchal
Copy link
Collaborator Author

OK. Agreed. Can you select a better full scale test then than the wasp?

I'll work on fixing the scaffolding merqury plot though to start in a new PR.

@mahesh-panchal mahesh-panchal merged commit f45b881 into main Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@MartinPippel MartinPippel MartinPippel approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mahesh-panchal @MartinPippel