Skip to content

[Variant] Support reading/writing Parquet Variant LogicalType #8370

@alamb

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Parquet include a Variant logical type that is used to tell other readers a column is a Variant:
https://github.com/apache/parquet-format/blob/c3f7be7dc615de76910154d690c534768ba25d6f/src/main/thrift/parquet.thrift#L494

We have been using Arrow extension types for the in memory manipulation of Variant.

We need to teach the rust reader / writer how to also use the Parquet LogicalType

Describe the solution you'd like
The rust writer should write the Variant logical type when writing VariantArrays

The rust reader should recognize columns of the Variant logical type and add the relevant arrow extension type metadata

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

Labels

enhancementAny new improvement worthy of a entry in the changelogparquetChanges to the parquet crateparquet-variantparquet-variant* crates

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions