You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parquet files - when compressed - are so in the page layer. Parquet supports compression per page, (as shown from the DataPageHeaderV2 IsCompressed field, which comes directly from the thrift definition). The library detects the compression type (called CompressionCodec) and passes that down to the newBlockReader level. However it still needs to check if that specific page is indeed compressed, and that was missing.
Unit test to reproduce
I have a slim and simple unit test here, but I could write a full-fledged one with a test file if required.
parquet-go specific details
v0.12.0
Misc Details
I have already patched this in a fork and we're using it in Panther's production for the last 2 weeks. It seems it's working.
I have tested it with a test file too (not sure where to upload it if you guys want it)
The text was updated successfully, but these errors were encountered:
Describe the bug
Parquet files - when compressed - are so in the page layer. Parquet supports compression per page, (as shown from the DataPageHeaderV2 IsCompressed field, which comes directly from the thrift definition). The library detects the compression type (called CompressionCodec) and passes that down to the newBlockReader level. However it still needs to check if that specific page is indeed compressed, and that was missing.
Unit test to reproduce
I have a slim and simple unit test here, but I could write a full-fledged one with a test file if required.
parquet-go specific details
v0.12.0
Misc Details
The text was updated successfully, but these errors were encountered: