Skip to content

Deprecate all non ArrowSession DType imports and exports - #8918

Open
robert3005 wants to merge 5 commits into
developfrom
rk/deprecate
Open

Deprecate all non ArrowSession DType imports and exports#8918
robert3005 wants to merge 5 commits into
developfrom
rk/deprecate

Conversation

@robert3005

@robert3005 robert3005 commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

We have a new interface that handles dynamically registering extensions, the
legacy methods shouldn't be expose as they don't properly delegate

@robert3005 robert3005 added the changelog/break A breaking API change label Jul 23, 2026
@codspeed-hq

codspeed-hq Bot commented Jul 23, 2026 •
edited
Loading

Copy link
Copy Markdown

Merging this PR will improve performance by 11.5%

⚡ 1 improved benchmark
✅ 1841 untouched benchmarks
⏩ 44 skipped benchmarks1

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Simulation compress_fsst[(1000, 64, 8)] 1.2 ms 1 ms +11.5%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.

Comparing rk/deprecate (a9a0291) with develop (85569a2)

Open in CodSpeed

Footnotes

  1. 44 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@robert3005 robert3005 mentioned this pull request Jul 23, 2026
Draft
robert3005 added a commit that referenced this pull request Jul 23, 2026
@robert3005 @claude
Deprecate FromArrowArray in favour of ArrowSession array imports
b9fb49f 
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 added a commit that referenced this pull request Jul 23, 2026
@robert3005 @claude
Deprecate FromArrowArray in favour of ArrowSession array imports
078e7b1 
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 added a commit that referenced this pull request Jul 23, 2026
@robert3005 @claude
Deprecate FromArrowArray in favour of ArrowSession array imports
8f7dc1c 
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 added a commit that referenced this pull request Jul 23, 2026
@robert3005 @claude
Deprecate FromArrowArray in favour of ArrowSession array imports
28f3268 
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
robert3005 added a commit that referenced this pull request Jul 31, 2026
@robert3005 @claude
Deprecate FromArrowArray in favour of ArrowSession array imports
ab2046a 
Follow-up to #8918: `FromArrowArray` is now #[deprecated] and every usage is
replaced with `ArrowSession` methods, making the session the authoritative
Arrow array import path.

- `ArrowSession` gains `from_arrow_array_nullable(&dyn Array, bool)` for
  imports with no Arrow `Field` in hand; nested fields still dispatch
  extension import plugins.
- `ArrowImportVTable::from_arrow_array` now receives the `ArrowSession` so
  plugins convert storage through the session (uuid, geo, json, tensor,
  parquet-variant updated).
- All consumers (datafusion, python, ffi, tui, bench, compat-gen, layout,
  facade doc snippet) migrated to `from_arrow_array` /
  `from_arrow_array_nullable` / `from_arrow_record_batch`.
- `vx_array_from_arrow` now imports through the session using the FFI
  schema's field, so Arrow extension types round-trip.
- `IntoArrowArray` was already deprecated and has no remaining users.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
onursatici
onursatici reviewed Aug 4, 2026
View reviewed changes
Comment thread vortex-arrow/src/session.rs Outdated
robert3005 and others added 5 commits August 5, 2026 11:49
@robert3005
Deprecate all non ArrowSession DType imports and exports
683d571 
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
fixes
853962a 
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 @claude
Thread ArrowSession explicitly instead of default-session statics
3455abe 
- vortex-datafusion: replace the `FromDataFusion<ScalarValue> for Scalar`
  impl with `scalar_from_df(value, &ArrowSession)`; the expression convertor
  passes its own session so nested extension columns resolve through the
  registered plugins.
- vortex-jni: `NativeDataSource`/`NativeScan` carry the session they were
  opened with; `export_dtype_to_arrow` takes the `ArrowSession`.
- vortex-ffi: `vx_dtype_to_arrow_schema` / `vx_dtype_from_arrow_schema` take
  a `const vx_session*` argument (header regenerated, example updated).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005 @claude
Update C++ API for session-taking dtype ⇄ Arrow schema conversions
804db7a 
DataType::from_arrow and DataType::to_arrow take a Session, matching the
new vx_dtype_from_arrow_schema / vx_dtype_to_arrow_schema signatures.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Robert Kruszewski <github@robertk.io>
@robert3005
Recurse into run-end values for extension types
a9a0291 
Arrow models run-end values as a Field, so it can carry extension metadata.
Both directions previously converted through the values `DataType` alone,
dropping that metadata: importing a run-end array of UUIDs produced the naive
storage dtype, and exporting to a run-end target failed outright.

Import now dispatches the values field through `from_arrow_field`, and
`from_arrow_array` grows a `RunEndEncoded` arm that recurses into the values
(previously run-end arrays were not importable through the generic path at all).
Export routes the values through the session rather than `execute_arrow_naive`.

Dictionary values stay on the naive mapping: Arrow models them as a bare
`DataType`, so they cannot carry extension metadata.

Signed-off-by: Robert Kruszewski <github@robertk.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@onursatici onursatici onursatici left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

changelog/break A breaking API change

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@robert3005 @onursatici