[DISCUSS] Iceberg V3 type support in Gravitino

[nevzheng]

Current state

While working on #11927 / #11928 (the native API failing to load Iceberg tables with a V3 variant column), I found that Gravitino's unified type model has no representation for Iceberg's V3 net-new types. With #11928 merged, all of them load through the native API as ExternalType:

Iceberg V3 type	Maps to today	Native Gravitino type?
variant	external(variant)	none
geometry	external(GEOMETRY)	none
geography	external(GEOGRAPHY)	none
timestamp_ns	external(TIMESTAMP_NANO)	looks possible — TimestampType takes a precision
timestamptz_ns	external(TIMESTAMP_NANO)	looks possible — TimestampType takes a precision
unknown	external(UNKNOWN)	maybe NullType — defer to maintainers

The question

Is this working-as-intended, or do we want native support for these types — and if so, how?

Decisions we need

1. Is ExternalType the intended long-term mapping for Iceberg V3 types, or is this a gap to close?
2. If we want native support — which types, and how do they enter the unified model (enum + JSON serde + DTOs + clients + per-catalog converters)?

Options to seed the discussion

• A — Keep V3 types as ExternalType. Document the limitations, defer native support. This is what ExternalType was designed for ([Subtask] Add ExternalType for unsolved catalog type  #3500). But note the downstream cost: external(...) columns aren't always consumable — e.g. external(serial) failed when queried through Trino ([Improvement] Unsupported Gravitino datatype: external(serial) when querying via Trino #10957), and the fix there was to give the type a native mapping rather than leave it external.
• B — Implement native support for V3 types. Add native types (e.g. VariantType, geospatial types) to the unified model. These are independent types (variant, geospatial, nanosecond timestamps), so they could be pursued separately / incrementally rather than as one change. timestamp_ns in particular looks close — TimestampType already carries precision ([Improvement] support for time and timestamp types with precision #2431).

References

How the escape hatches were designed

• [Subtask] Add ExternalType for unsolved catalog type  #3500 / [#3500] feat(relational): add ExternalType data type #3501 — ExternalType added for "unsolved catalog type" (its intended purpose).
• [Improvement] Add Unknown column data type to handle an unresolvable type from the catalog #2117 / [#2117] improvement(api): Type adds UNPARSED column data type to handle an unresolvable type from the catalog #2140 — UnparsedType added to handle an unresolvable type from the catalog (the alternative fallback).

Where ExternalType has caused problems

• [Improvement] Unsupported Gravitino datatype: external(serial) when querying via Trino #10957 / [#10957] fix(catalog-postgresql): Map serial types to integer types in PostgreSqlTypeConverter #10970 — external(serial) unqueryable via Trino; resolved by mapping to a native type.
• [Improvement]: ExternalType.catalogString() is written verbatim into DDL SQL without validation in JDBC catalog #11805 — ExternalType.catalogString() written verbatim into DDL SQL (write-path hazard).
• [Bug] ClickHouse catalog: GraphiteMergeTree misclassified and multiple types fall through to ExternalType #11910 / [#11910] fix(clickhouse): fix GraphiteMergeTree classification and add missing type mappings #11915 — ClickHouse types falling through to ExternalType.

Iceberg V3 surface

• [Bug report] Iceberg native metadata API fails to load tables with V3 variant columns #11927 / [#11927] fix(iceberg): load tables containing V3 variant columns #11928 — the variant load bug + fix that surfaced this.
• [Improvement] Support Iceberg V3 variant type in JDBC catalog backend of Iceberg REST Server #10994 — variant on the IRC/JDBC backend; noted as under a dedicated community meeting.
• [FEATURE] Upgrade server-side Apache Iceberg to 1.11.0 for IRC and lakehouse catalog #11293 / [#11293] feat(iceberg): Upgrade server-side Iceberg to 1.11.0 for REST service and lakehouse catalog #11219 — server-side Iceberg upgraded to 1.11.0 (brought V3 types into range).
• [Bug report] Deletion vectors are not written when deleting rows from an Iceberg v3 table via Gravitino IRC #10995, [Bug report] Trino creates a table with format_version = 3 for a remote IRC, but the resulting table ends up with format_version = 2 #10986 — other V3 gaps (deletion vectors; format-version-3 downgraded to 2).

Timestamp precision precedent

• [Improvement] support for time and timestamp types with precision #2431 — time/timestamp precision support (why timestamp_ns looks representable natively).

---

cc @roryqi @markhoerth — can you help me find the right people to pull into this thread? Thanks!

[jerryshao]

I think we should add a built-in variant type support in Gravitino.

[roryqi]

Maybe we can fix the error types in version 1.3 using external type. We can add new built-in type in version 2.0. Just my cent.

[xxubai]

I can help with the Iceberg V3 adaptation, such as supporting built-in types like Variant :)

[nevzheng]

• I got the go ahead to own + drive discussion + implementation of Iceberg V3 support. Starting with variant.

[nevzheng]

Proposal: Native variant type support in Gravitino

Status: draft / notes

Summary

Promote Iceberg V3 variant from the ExternalType("variant") stopgap (#11928) to a
first-class VariantType leaf. It stays opaque and just gains a real identity in the
type model. Small change: at the schema level a variant is a parameterless leaf, the
same shape as the existing UUIDType.

What a variant is

A column type for arbitrary, JSON-like semi-structured data whose shape varies row to
row. At the schema level it is the bare, parameterless word variant: all structure
lives in the values, not the type. A metadata catalog sees one opaque leaf.

Current state

Gravitino only records that a column is a variant; it never reads values. The IRC path
already handled it (#10994); the native converter (#11927) threw UnsupportedOperationException: Unsupported type: variant. After #11928 it maps to ExternalType.of("variant"), so the
column loads. That is already a relabeled blob, but as a string, not a type: nothing can
branch on it or map it deliberately per connector.

The change

Serde here is of the schema, not values: a Type object round-trips to JSON to cross the
REST API and the entity store. A variant leaf serializes to the bare token "variant"
(like uuid), so this is four small edits:

• Type.Name.VARIANT - new enum tag.
• Types.VariantType - singleton leaf extending PrimitiveType (a UUIDType clone).
• JsonUtils - add VariantType.get() to the primitive TYPES map (read) and a
  case VARIANT in writeDataType (write).
• FromIcebergType.variant() - return Types.VariantType.get() instead of ExternalType.

Out of scope: introspection (no sub-fields, no variant_get-style accessors). Those are
query-engine operations on row data, not catalog metadata.

Decisions and trade-offs

• Extend PrimitiveType (not ComplexType, not raw Type): a parameterless childless
  leaf fits the primitive serde path with no bespoke code. Risk: "primitive" reads oddly
  for semi-structured data, but the base class classifies by structure and uuid/binary
  set the precedent.
• Iceberg-first: other connectors keep mapping variant to ExternalType. Adding
  Name.VARIANT touches every Type.Name switch (Hive, Glue, Paimon, Lance, Trino,
  Flink, JDBC, Spark), but only Iceberg emits one today. Risk: a new constant can reach a
  switch default elsewhere; audit each and convert connectors individually later.
• Why not stay external: external(serial) is unqueryable via Trino ([Improvement] Unsupported Gravitino datatype: external(serial) when querying via Trino #10957) and
  catalogString() has been written verbatim into DDL SQL unvalidated ([Improvement]: ExternalType.catalogString() is written verbatim into DDL SQL without validation in JDBC catalog #11805). A modeled
  identity avoids passing an opaque string through those paths.

PoC

#11932

References

• [Bug report] Iceberg native metadata API fails to load tables with V3 variant columns #11927 (open, bug) - native metadata API fails to load V3 variant columns.
• [#11927] fix(iceberg): load tables containing V3 variant columns #11928 (open, PR) - the ExternalType stopgap this builds on.
• [Improvement] Support Iceberg V3 variant type in JDBC catalog backend of Iceberg REST Server #10994 (closed) - V3 variant in the Iceberg REST server JDBC backend (the IRC path).
• [Subtask] Add ExternalType for unsolved catalog type  #3500 (closed) - ExternalType for unsolved catalog type (the escape hatch's origin).
• [Improvement] Unsupported Gravitino datatype: external(serial) when querying via Trino #10957 (open) - external(serial) unqueryable via Trino.
• [Improvement]: ExternalType.catalogString() is written verbatim into DDL SQL without validation in JDBC catalog #11805 (open) - ExternalType.catalogString() written verbatim into DDL SQL.

[roryqi]

Thanks your proposal. Maybe you write a design document and raise a pull request. You can see #11852
You can refer to our design document skills https://github.com/apache/gravitino/tree/main/.claude/skills/gravitino-design-doc .

[jerryshao]

@mchades Can you please also take a look? BTW, this is a small feature, I don't think we need a formal design doc for this.

[mchades]

I am reviewing PR #11932 and suggest that you create an issue from this discussion to formally track the matter.

[nevzheng]

Acknowledged — #11949 is now the canonical issue tracking native Iceberg V3 variant support, implemented by #11932. Let's focus further discussion there; this thread stays as background. Thanks all!