Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multiprocessed fastq output #65

Merged
merged 6 commits into from
Mar 21, 2024
Merged

Conversation

remiolsen
Copy link
Member

@remiolsen remiolsen commented Jan 25, 2024

closes #3

Parallelise per fastq file output (and not fastq file entries chunks). I will not explain further.

Needs some testing. Setting this as draft PR

@remiolsen remiolsen force-pushed the parallel_fastq_out branch from 5b0ef26 to e119808 Compare March 6, 2024 10:09
@remiolsen remiolsen marked this pull request as ready for review March 6, 2024 12:51
@remiolsen
Copy link
Member Author

remiolsen commented Mar 6, 2024

Also updated to ruff 0.3.0 for ruff-precommit, since the old one was fighting with GH action's version of ruff.

@remiolsen remiolsen requested a review from alneberg March 6, 2024 14:47
@remiolsen remiolsen marked this pull request as draft March 6, 2024 15:25
@remiolsen
Copy link
Member Author

I changed my mind. I might as well try to parallise lenient mode while I'm at it.

@remiolsen
Copy link
Member Author

closes #50

I'm satisfied that my multiprocessing.spawn usage works on OSX and Linux (Miarka). Hopefully it will not cause Windows to explode and catch fire.

Ready for review now @alneberg

@remiolsen remiolsen marked this pull request as ready for review March 13, 2024 12:46
Copy link
Member

@alneberg alneberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, and runs fine both on the m2 macbook and linux server.

@remiolsen remiolsen merged commit 762d826 into master Mar 21, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Multi-thread fastq read/write operations
2 participants