New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet: add variant type support #11653

Draft

aihuaxu wants to merge 1 commit into apache:main from aihuaxu:variant-type-parquet

Contributor

aihuaxu commented Nov 26, 2024 •

edited

Loading

This is to enable writing Variant data to Parquet file and read from Parquet file in the format of struct of metadata and data.

This PR adds:

A Parquet Variant implementation which currently uses Spark-Variant library as the dependency. We can reimplement inside the Iceberg-Core library to remove such dependency for now before that is moved to Parquet project.
Add necessary Variant support in Parquet reader/writer.

TODO:

Add more test coverage.
Implement Variant encoding/decoding to remove Spark-Variant library dependency.
Add Parquet logical type Variant so ParquetTypeVisitor can understand Variant type from Parquet annotation.

Fix: #11178.


          Parquet: add variant type support

09b7fe3

github-actions bot added API parquet arrow core build labels

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

API arrow build core parquet