add offset header ahead read for chunks

[bhimrazy]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Add offset header ahead loading for chunks in PyTreeLoader

Some test metrics:

Note: These were generated locally on a certain set of random fake data

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

[codecov]

Codecov Report

Attention: Patch coverage is 95.45455% with 1 line in your changes missing coverage. Please review.

Project coverage is 79%. Comparing base (ba4e981) to head (75963b2).

Additional details and impacted files

@@         Coverage Diff         @@
##           main   #501   +/-   ##
===================================
  Coverage    79%    79%           
===================================
  Files        38     38           
  Lines      5659   5679   +20     
===================================
+ Hits       4483   4502   +19     
- Misses     1176   1177    +1     

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tchaton]

Mind adding the benchmark to the description of the PR ?

[bhimrazy]

After benchmarking on ImageNet(resized 13GB), this PR's changes (header ahead read approach) do not seem to make any significant difference on a large dataset.

[bhimrazy]

Benchmark results on Imagenet (137GB)

Original v0.2.41

PR 501

[tchaton]

@bhimrazy So it doesn't seem very significant for larger dataset where more time is allocated to data transformation but it might be smaller ones with lower data transformation ?

[bhimrazy]

@bhimrazy So it doesn't seem very significant for larger dataset where more time is allocated to data transformation but it might be smaller ones with lower data transformation ?

Maybe but I think it would be good to test on tiny imagenet too and also check the profiling.