Epic: `ScanPlan` (replacing `LayoutReader`s)

[joseph-isaacs]

This Epic is for tracking the migration from LayoutReader to Plan to new system similar to Arrays for reducing and executing arrays. Plans need compute and I/O and maybe subtasks. For now we will move the reduce rules. This could be seen as a physical plan (Plan) and execution will be executing that plan.

We currently have a LayoutReader which operates on expressions using filter_evaluation, projection_evaluation etc. These are given an expression which they pass split up and pass into children LayoutReader. A ScanPlan will act similarly to ArrayRef

Idea

Lets say we have a Layout

StructLayout(a: FlatLayout, b: FlatLayout) and we want to apply the expr $.a + 1 the struct layout reader would be passed that expr partition the expression request $+1 from the a: FlatLayout and return it.

In the Plan system we would create

ExprPlan(
  expr: $.a+1
  child:
    StructPlan(
    a: FlatLayout
    b: FlatLayout
  )
)

which would be optimised to

p := ExprPlan(
  expr: $+1
  child: FlatLayout // flat layout reader from struct field a
)

p would then be evaluated using some out of scope execution system. (For now we will use the LayoutReader system we have in vortex)

Status

• Add a ScanPlan trait
• Add a ExpressionScanPlan Tracking Issue: ExpressionPlan #9115
• Add custom pushdown for ExpressionScanPlan x ScanPlan (Struct, Dict, etc...)

Goal

• Move to Scan from using LayoutReader API to ScanPlan API
• Find and use a new execution system for Scan Plans. This Layout Reader V27 #8954 or something else.

Motivation

• This will give us control of expression/aggregation and other pushdown.
• The changed executor will give us control over resource usage.

Unresolved questions

• None yet.