-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Parquet include a Variant logical type that is used to tell other readers a column is a Variant:
https://github.com/apache/parquet-format/blob/c3f7be7dc615de76910154d690c534768ba25d6f/src/main/thrift/parquet.thrift#L494
We have been using Arrow extension types for the in memory manipulation of Variant.
We need to teach the rust reader / writer how to also use the Parquet LogicalType
Describe the solution you'd like
The rust writer should write the Variant logical type when writing VariantArrays
The rust reader should recognize columns of the Variant logical type and add the relevant arrow extension type metadata
Describe alternatives you've considered
Additional context