Schema Evolution Support in Velox

[Yuhta]

Currently the schema evolution support in Velox is very limited: it only supports positional matching of columns and subfields based on the usage of data warehouse inside Meta, and only implemented for DWRF reader. We will need to extend the functionality to support more use cases:

1. Match by position
2. Match by field name (e.g. Gluten)
3. Match by unique permanent ID (e.g. Iceberg)

Thus here I start a discussion to collect the information first and will put out designs to be discuss.

@rui-mo Can you share some detail about how schema evolution is supposed to be handled in Gluten/Spark?

[rui-mo]

@Yuhta Below are some details.
Spark now supports both matching by ID (since v3.3.0) and by name. As quoted from Spark,

when the requested schema has field IDs, Parquet readers will first use the field ID to determine which Parquet columns to read if the field ID exists in Spark schema, before falling back to match using column names.

The config spark.sql.parquet.fieldId.read.enabled is false by default, so matching by name is still the default behavior.

Links:
To support matching by ID: apache/spark@b5eae59, and please check ParquetRowConverter.
Previously, only matching by name: apache/spark@215609d

[Yuhta]

Thanks for the info. One question, where do we keep the ID inside parquet file? Is it in user metadata?

[rui-mo]

Yes, it is in the SchemaElement under file metadata, and please check:
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift#L398-L401.