Code

Parallelize I/O operations#97

Code

Merged

jeromekelleher merged 14 commits intosgkit-dev:mainsgkit-dev/vcztools:mainfrom Will-Tyler:parallelWill-Tyler/vcztools:parallelCopy head branch name to clipboardNov 19, 2024

+119-84

Commits

Commits on Nov 6, 2024

• Parallelize chunk loading

  Will-Tyler

  committed
  569f83b

• Parallelize chunk writing

  Will-Tyler

  committed
  542e5ca

• Wait for preceding future before writing

  Will-Tyler

  committed
  c5d9a2e

• Add multichunk test dataset

  Will-Tyler

  committed
  93fc658

• Add multichunk validation test

  Will-Tyler

  committed
  686682e

• Release GIL while encoding VCF

  Will-Tyler

  committed
  0a47519

• Load data on same thread

  Will-Tyler

  committed
  e122ecc

• Move waiting to just before writing

  Will-Tyler

  committed
  15642cb

• Push variant-wise for-loop to C extension

  Will-Tyler

  committed
  e62c497

• Use pipe when output file descriptor is not available

  Will-Tyler

  committed
  2d21289

• Revert "Use pipe when output file descriptor is not available"

  Show description for ca43b16
  Will-Tyler

  committed
  ca43b16

• Revert "Push variant-wise for-loop to C extension"

  Show description for 989f701
  Will-Tyler

  committed
  989f701

• Fix ruff error

  Will-Tyler

  committed
  3fa5229

• Revert "Release GIL while encoding VCF"

  Show description for 274ef0a
  Will-Tyler

  committed
  274ef0a