Reference Tests and Breaking Change: Optional nullable fields are now null instead of undefined #114
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wilwade
commented
Jan 18, 2024
Comment on lines
+27
to
+28
| 'float16_nonzeros_and_nans.parquet', // missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY) | ||
| 'float16_zeros_and_nans.parquet', // missing option: typeLength (required for FIXED_LEN_BYTE_ARRAY) |
There was a problem hiding this comment.
These can be removed once #108 is rebased on it and will work as a test to prove it is working.
wilwade
commented
Jan 18, 2024
Comment on lines
+230
to
+231
| // This ensures that nulls are correctly processed | ||
| record[node.name] = value; |
There was a problem hiding this comment.
This is the change that makes null values shred to null instead of undefined.
wilwade
commented
Jan 18, 2024
Comment on lines
+675
to
+677
| // If this exists and is greater than zero then we need to have an offset | ||
| if (metadata.dictionary_page_offset && +metadata.dictionary_page_offset > 0) { | ||
| const offset: number = +metadata.dictionary_page_offset; |
There was a problem hiding this comment.
Based in part around the information in https://issues.apache.org/jira/browse/PARQUET-1402
wilwade
commented
Jan 18, 2024
There was a problem hiding this comment.
Fixes for using null instead of undefined
wilwade
commented
Jan 18, 2024
There was a problem hiding this comment.
Fixes for using null instead of undefined
wilwade
commented
Jan 18, 2024
There was a problem hiding this comment.
Fixes for using null instead of undefined
shannonwells
approved these changes
Jan 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We wanted to add tests for all the tests in https://github.com/apache/parquet-testing
Discovered Bugs
Solution
test/reference-test/read-all.test.tswith @shannonwells
Steps to Verify: