Allow reading Parquet maps that lack a values field
#6730
Conversation
There was a problem hiding this comment.
This looks good to me @etseidl -- thanky ou
I also poked around in https://github.com/apache/parquet-testing/tree/master/data for an example of such a file, but it seems like we do not have one.
| let out = reader.next().unwrap().unwrap(); | ||
| assert_eq!(out.num_rows(), 3); | ||
| assert_eq!(out.num_columns(), 2); | ||
| // map and list columns should now be equivalent |
There was a problem hiding this comment.
So this means the parquet reader will read MAP without values as an ListArray -- I think this seems like the intention so 👍
There was a problem hiding this comment.
Yes, because an arrow map must have non-null values IIUC.
Correct, which is why the effort in the tests to produce one. I'll try submitting one I have on hand to parquet-testing. |
|
apache/parquet-testing#63 submitted. Maybe we can hold off on merging this to see if the test file will be accepted. |
Which issue does this PR close?
Closes #1642.
Rationale for this change
The Parquet spec does not require the
valuesfield of a map to be present, but current readers will error out if this field is missing.What changes are included in this PR?
Changes both the record reader and arrow reader to read a
MAPlackingvaluesas a list of keys. This matches the behavior of arrow-cpp.Are there any user-facing changes?
No