Skip to content

Add plan-native scan execution - #9240

Draft
joseph-isaacs wants to merge 1 commit into
vortex-plan-zonedfrom
vortex-plan-scan-v2
Draft

Add plan-native scan execution#9240
joseph-isaacs wants to merge 1 commit into
vortex-plan-zonedfrom
vortex-plan-scan-v2

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add a separate vortex-scan-v2 crate copied from the existing scan orchestration and backed
    only by physical plans
  • bind raw projection and filter expressions once at the scan-builder boundary, then optimize and
    execute only BoundExpression plan nodes
  • add plan execution for flat, chunked, dictionary, struct, list, row-index, expression, and zoned
    nodes; expression execution uses apply_bound
  • add an execution regression proving a reduced StructPlan never requests an absent unused-field
    segment
  • add plan-native split discovery plus debug/trace logging for source layouts, optimized plans,
    split ranges, and task results
  • add synthetic end-to-end tests and a TPC-H example that compares every result value with the
    existing LayoutReader scan

This keeps the original scan path unchanged while establishing the smallest independent path for
scanning through plans.

Current scope

  • zoned plans execute their data child but do not perform zone-map pruning yet
  • the existing scanner's dynamic conjunct ordering, reader caching, and pruning behavior are not
    copied into this correctness-first path

Validation

  • cargo +nightly fmt --all -- --check
  • RUSTC_WRAPPER= cargo clippy -p vortex-layout -p vortex-scan-v2 --all-targets --all-features -- -D warnings
  • RUSTC_WRAPPER= cargo test -p vortex-layout plan:: — 15 passed
  • RUSTC_WRAPPER= cargo test -p vortex-scan-v2 — 3 passed
  • RUSTC_WRAPPER= cargo test --doc -p vortex-layout -p vortex-scan-v2
  • git diff --check
  • RUSTC_WRAPPER= RUST_LOG=warn cargo run -p vortex-scan-v2 --example tpch_scan -- vortex-bench/data/tpch/0.01/vortex-file-compressed/lineitem.vortex
    • optimized the 16-field input to a two-field projection plan through the base optimizer PR
    • scanned 60,175 input rows and returned 6,494 rows
    • validated every result value against the existing LayoutReader scan

Stacked on #9187.

@codspeed-hq

codspeed-hq Bot commented Aug 6, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

✅ 1929 untouched benchmarks
⏩ 51 skipped benchmarks1


Comparing vortex-plan-scan-v2 (fba0962) with vortex-plan-zoned (24bf2c8)

Open in CodSpeed

Footnotes

  1. 51 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Give `PlanVTable` an `execute` hook taking a row range and a selection
mask, and implement it for every operator: `SegmentScan` reads and decodes
its segment, the structural operators combine their children, and `Eval`
applies the expression to its child's output.

Add `vortex-scan-v2`, which copies the existing scan orchestration around
this API so the `LayoutReader` scanner is untouched while the plan-native
path is developed.

`Take` now records whether every dictionary value is referenced by some
code. That fact previously came from the dict layout; since operators no
longer hold a layout, lowering carries it into the operator's data.

Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012obBhJ8oPZoBbKyeS79yMv
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/feature A new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant