Skip to content

Conversation

@Dandandan
Copy link
Contributor

@Dandandan Dandandan commented Nov 11, 2025

Which issue does this PR close?

Rationale for this change

Moe coalesce batches inside filter exec. We can use BatchCoalescer ::push_batch_with_filter which should give a speed up compared to filtering individual batches + concatenating afterwards.

What changes are included in this PR?

Changes the FilterExec to coalesce batches inside.
I did not make a change to remove CoalesceBatchesExec from all the plans, I plan to create an issue and a PR after this is merged to do so.
Now it should be mostly a no-op with limited overhead as input batches are already well-sized.

Are these changes tested?

Are there any user-facing changes?

@github-actions github-actions bot added core Core DataFusion crate physical-plan Changes to the physical-plan crate labels Nov 11, 2025
@Dandandan
Copy link
Contributor Author

@alamb could you run some benchmarks?

@alamb
Copy link
Contributor

alamb commented Nov 11, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (4386c68) to 9029ff1 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very nice 👌 I kicked off some benchmarks so let's see how that goes

continue;
}
None => {
self.batch_coalescer.finish_buffered_batch().unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should probably check for the error here rather than unwrapping

@alamb
Copy link
Contributor

alamb commented Nov 11, 2025

🤖: Benchmark completed

Details

Comparing HEAD and coalesce_batches_filter
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ coalesce_batches_filter ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  2691.53 ms │              2659.32 ms │ no change │
│ QQuery 1     │  1253.57 ms │              1225.87 ms │ no change │
│ QQuery 2     │  2337.37 ms │              2365.69 ms │ no change │
│ QQuery 3     │  1212.27 ms │              1178.42 ms │ no change │
│ QQuery 4     │  2320.18 ms │              2315.67 ms │ no change │
│ QQuery 5     │ 28101.95 ms │             28087.34 ms │ no change │
│ QQuery 6     │  4054.98 ms │              4040.75 ms │ no change │
│ QQuery 7     │  3658.92 ms │              3553.58 ms │ no change │
└──────────────┴─────────────┴─────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 45630.78ms │
│ Total Time (coalesce_batches_filter)   │ 45426.64ms │
│ Average Time (HEAD)                    │  5703.85ms │
│ Average Time (coalesce_batches_filter) │  5678.33ms │
│ Queries Faster                         │          0 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │          8 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ coalesce_batches_filter ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.46 ms │                 2.65 ms │  1.08x slower │
│ QQuery 1     │    49.87 ms │                49.22 ms │     no change │
│ QQuery 2     │   137.03 ms │               138.01 ms │     no change │
│ QQuery 3     │   163.67 ms │               160.12 ms │     no change │
│ QQuery 4     │  1043.67 ms │              1207.21 ms │  1.16x slower │
│ QQuery 5     │  1485.47 ms │              1527.46 ms │     no change │
│ QQuery 6     │     2.20 ms │                 2.13 ms │     no change │
│ QQuery 7     │    53.74 ms │                57.35 ms │  1.07x slower │
│ QQuery 8     │  1447.65 ms │              1536.81 ms │  1.06x slower │
│ QQuery 9     │  1813.05 ms │              1978.63 ms │  1.09x slower │
│ QQuery 10    │   377.10 ms │               372.54 ms │     no change │
│ QQuery 11    │   419.98 ms │               430.20 ms │     no change │
│ QQuery 12    │  1342.22 ms │              1475.60 ms │  1.10x slower │
│ QQuery 13    │  2120.69 ms │              2203.66 ms │     no change │
│ QQuery 14    │  1245.76 ms │              1331.51 ms │  1.07x slower │
│ QQuery 15    │  1223.22 ms │              1302.32 ms │  1.06x slower │
│ QQuery 16    │  2697.72 ms │              2714.05 ms │     no change │
│ QQuery 17    │  2655.54 ms │              2698.10 ms │     no change │
│ QQuery 18    │  5465.26 ms │              5014.92 ms │ +1.09x faster │
│ QQuery 19    │   127.98 ms │               130.55 ms │     no change │
│ QQuery 20    │  2048.52 ms │              1989.80 ms │     no change │
│ QQuery 21    │  2372.60 ms │              2343.35 ms │     no change │
│ QQuery 22    │  3980.08 ms │              3992.99 ms │     no change │
│ QQuery 23    │ 13374.23 ms │             12881.66 ms │     no change │
│ QQuery 24    │   218.86 ms │               209.13 ms │     no change │
│ QQuery 25    │   473.30 ms │               482.03 ms │     no change │
│ QQuery 26    │   223.91 ms │               222.54 ms │     no change │
│ QQuery 27    │  2822.85 ms │              2841.47 ms │     no change │
│ QQuery 28    │ 23428.70 ms │             23252.67 ms │     no change │
│ QQuery 29    │   991.04 ms │               960.29 ms │     no change │
│ QQuery 30    │  1324.59 ms │              1354.69 ms │     no change │
│ QQuery 31    │  1393.96 ms │              1368.83 ms │     no change │
│ QQuery 32    │  4845.05 ms │              4577.94 ms │ +1.06x faster │
│ QQuery 33    │  5940.04 ms │              5771.19 ms │     no change │
│ QQuery 34    │  6245.80 ms │              5973.56 ms │     no change │
│ QQuery 35    │  2064.77 ms │              2089.54 ms │     no change │
│ QQuery 36    │   125.06 ms │               124.77 ms │     no change │
│ QQuery 37    │    52.92 ms │                53.77 ms │     no change │
│ QQuery 38    │   120.71 ms │               122.10 ms │     no change │
│ QQuery 39    │   197.44 ms │               196.82 ms │     no change │
│ QQuery 40    │    42.02 ms │                44.08 ms │     no change │
│ QQuery 41    │    39.13 ms │                38.90 ms │     no change │
│ QQuery 42    │    32.83 ms │                33.40 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 96232.70ms │
│ Total Time (coalesce_batches_filter)   │ 95258.55ms │
│ Average Time (HEAD)                    │  2237.97ms │
│ Average Time (coalesce_batches_filter) │  2215.32ms │
│ Queries Faster                         │          2 │
│ Queries Slower                         │          8 │
│ Queries with No Change                 │         33 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ coalesce_batches_filter ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 1     │ 137.27 ms │               138.89 ms │     no change │
│ QQuery 2     │  29.24 ms │                28.40 ms │     no change │
│ QQuery 3     │  37.97 ms │                38.59 ms │     no change │
│ QQuery 4     │  29.81 ms │                29.05 ms │     no change │
│ QQuery 5     │  87.56 ms │                88.26 ms │     no change │
│ QQuery 6     │  20.04 ms │                19.77 ms │     no change │
│ QQuery 7     │ 224.05 ms │               229.62 ms │     no change │
│ QQuery 8     │  33.29 ms │                32.85 ms │     no change │
│ QQuery 9     │ 103.80 ms │               107.28 ms │     no change │
│ QQuery 10    │  64.03 ms │                64.83 ms │     no change │
│ QQuery 11    │  18.83 ms │                16.69 ms │ +1.13x faster │
│ QQuery 12    │  50.86 ms │                52.40 ms │     no change │
│ QQuery 13    │  47.28 ms │                44.90 ms │ +1.05x faster │
│ QQuery 14    │  13.84 ms │                13.97 ms │     no change │
│ QQuery 15    │  25.09 ms │                25.16 ms │     no change │
│ QQuery 16    │  24.95 ms │                25.44 ms │     no change │
│ QQuery 17    │ 148.54 ms │               151.79 ms │     no change │
│ QQuery 18    │ 282.62 ms │               279.84 ms │     no change │
│ QQuery 19    │  38.34 ms │                38.26 ms │     no change │
│ QQuery 20    │  49.08 ms │                50.09 ms │     no change │
│ QQuery 21    │ 321.30 ms │               327.13 ms │     no change │
│ QQuery 22    │  20.72 ms │                20.44 ms │     no change │
└──────────────┴───────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1808.52ms │
│ Total Time (coalesce_batches_filter)   │ 1823.66ms │
│ Average Time (HEAD)                    │   82.21ms │
│ Average Time (coalesce_batches_filter) │   82.89ms │
│ Queries Faster                         │         2 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        20 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

@alamb
Copy link
Contributor

alamb commented Nov 11, 2025

🤖 ./gh_compare_branch.sh Benchmark Script Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing coalesce_batches_filter (c11583d) to 9029ff1 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb
Copy link
Contributor

alamb commented Nov 11, 2025

The lack of consistency is really bugging me -- I am going to see if I can find some way to have more consistent benchmarks

@alamb
Copy link
Contributor

alamb commented Nov 11, 2025

🤖: Benchmark completed

Details

Comparing HEAD and coalesce_batches_filter
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ coalesce_batches_filter ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │  2642.14 ms │              2599.20 ms │     no change │
│ QQuery 1     │  1262.40 ms │              1190.50 ms │ +1.06x faster │
│ QQuery 2     │  2416.97 ms │              2337.11 ms │     no change │
│ QQuery 3     │  1210.38 ms │              1190.89 ms │     no change │
│ QQuery 4     │  2297.50 ms │              2288.48 ms │     no change │
│ QQuery 5     │ 28265.49 ms │             28202.83 ms │     no change │
│ QQuery 6     │  4085.13 ms │              4058.92 ms │     no change │
│ QQuery 7     │  3573.53 ms │              3606.53 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 45753.53ms │
│ Total Time (coalesce_batches_filter)   │ 45474.45ms │
│ Average Time (HEAD)                    │  5719.19ms │
│ Average Time (coalesce_batches_filter) │  5684.31ms │
│ Queries Faster                         │          1 │
│ Queries Slower                         │          0 │
│ Queries with No Change                 │          7 │
│ Queries with Failure                   │          0 │
└────────────────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ coalesce_batches_filter ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.54 ms │                 2.55 ms │     no change │
│ QQuery 1     │    49.83 ms │                50.92 ms │     no change │
│ QQuery 2     │   136.99 ms │               135.65 ms │     no change │
│ QQuery 3     │   164.01 ms │               164.64 ms │     no change │
│ QQuery 4     │  1056.37 ms │              1063.00 ms │     no change │
│ QQuery 5     │  1455.77 ms │              1531.93 ms │  1.05x slower │
│ QQuery 6     │     2.16 ms │                 2.17 ms │     no change │
│ QQuery 7     │    55.59 ms │                55.77 ms │     no change │
│ QQuery 8     │  1469.91 ms │              1465.92 ms │     no change │
│ QQuery 9     │  1875.69 ms │              1775.66 ms │ +1.06x faster │
│ QQuery 10    │   370.91 ms │               371.31 ms │     no change │
│ QQuery 11    │   431.72 ms │               432.71 ms │     no change │
│ QQuery 12    │  1338.02 ms │              1370.48 ms │     no change │
│ QQuery 13    │  2082.60 ms │              2151.20 ms │     no change │
│ QQuery 14    │  1253.94 ms │              1283.32 ms │     no change │
│ QQuery 15    │  1218.88 ms │              1240.67 ms │     no change │
│ QQuery 16    │  2667.64 ms │              2691.51 ms │     no change │
│ QQuery 17    │  2651.76 ms │              2683.53 ms │     no change │
│ QQuery 18    │  5403.64 ms │              5005.25 ms │ +1.08x faster │
│ QQuery 19    │   130.78 ms │               127.08 ms │     no change │
│ QQuery 20    │  2037.08 ms │              1993.61 ms │     no change │
│ QQuery 21    │  2336.77 ms │              2286.01 ms │     no change │
│ QQuery 22    │  4035.61 ms │              3968.79 ms │     no change │
│ QQuery 23    │ 17891.15 ms │             12842.71 ms │ +1.39x faster │
│ QQuery 24    │   227.55 ms │               216.66 ms │     no change │
│ QQuery 25    │   480.59 ms │               466.63 ms │     no change │
│ QQuery 26    │   218.85 ms │               213.88 ms │     no change │
│ QQuery 27    │  2874.47 ms │              2808.06 ms │     no change │
│ QQuery 28    │ 23827.66 ms │             23394.83 ms │     no change │
│ QQuery 29    │   997.09 ms │               997.10 ms │     no change │
│ QQuery 30    │  1376.95 ms │              1320.52 ms │     no change │
│ QQuery 31    │  1413.16 ms │              1392.63 ms │     no change │
│ QQuery 32    │  5318.38 ms │              4522.89 ms │ +1.18x faster │
│ QQuery 33    │  6167.75 ms │              5846.51 ms │ +1.05x faster │
│ QQuery 34    │  6174.07 ms │              6017.32 ms │     no change │
│ QQuery 35    │  2044.10 ms │              2060.33 ms │     no change │
│ QQuery 36    │   123.68 ms │               123.54 ms │     no change │
│ QQuery 37    │    51.76 ms │                53.65 ms │     no change │
│ QQuery 38    │   121.67 ms │               121.80 ms │     no change │
│ QQuery 39    │   202.62 ms │               204.62 ms │     no change │
│ QQuery 40    │    42.76 ms │                42.50 ms │     no change │
│ QQuery 41    │    39.82 ms │                40.41 ms │     no change │
│ QQuery 42    │    33.26 ms │                32.75 ms │     no change │
└──────────────┴─────────────┴─────────────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃             ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 101855.58ms │
│ Total Time (coalesce_batches_filter)   │  94573.03ms │
│ Average Time (HEAD)                    │   2368.73ms │
│ Average Time (coalesce_batches_filter) │   2199.37ms │
│ Queries Faster                         │           5 │
│ Queries Slower                         │           1 │
│ Queries with No Change                 │          37 │
│ Queries with Failure                   │           0 │
└────────────────────────────────────────┴─────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ coalesce_batches_filter ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 140.56 ms │               137.18 ms │ no change │
│ QQuery 2     │  29.20 ms │                29.32 ms │ no change │
│ QQuery 3     │  38.93 ms │                38.67 ms │ no change │
│ QQuery 4     │  29.81 ms │                29.69 ms │ no change │
│ QQuery 5     │  87.50 ms │                88.18 ms │ no change │
│ QQuery 6     │  19.94 ms │                19.80 ms │ no change │
│ QQuery 7     │ 217.31 ms │               221.90 ms │ no change │
│ QQuery 8     │  34.71 ms │                33.63 ms │ no change │
│ QQuery 9     │  97.86 ms │               101.97 ms │ no change │
│ QQuery 10    │  64.26 ms │                64.10 ms │ no change │
│ QQuery 11    │  18.00 ms │                17.53 ms │ no change │
│ QQuery 12    │  51.96 ms │                50.71 ms │ no change │
│ QQuery 13    │  46.71 ms │                48.18 ms │ no change │
│ QQuery 14    │  13.98 ms │                14.07 ms │ no change │
│ QQuery 15    │  24.68 ms │                24.97 ms │ no change │
│ QQuery 16    │  25.44 ms │                25.25 ms │ no change │
│ QQuery 17    │ 154.66 ms │               152.63 ms │ no change │
│ QQuery 18    │ 281.57 ms │               276.58 ms │ no change │
│ QQuery 19    │  38.52 ms │                38.92 ms │ no change │
│ QQuery 20    │  48.75 ms │                49.09 ms │ no change │
│ QQuery 21    │ 305.96 ms │               317.17 ms │ no change │
│ QQuery 22    │  20.50 ms │                20.94 ms │ no change │
└──────────────┴───────────┴─────────────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary                      ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)                      │ 1790.83ms │
│ Total Time (coalesce_batches_filter)   │ 1800.46ms │
│ Average Time (HEAD)                    │   81.40ms │
│ Average Time (coalesce_batches_filter) │   81.84ms │
│ Queries Faster                         │         0 │
│ Queries Slower                         │         0 │
│ Queries with No Change                 │        22 │
│ Queries with Failure                   │         0 │
└────────────────────────────────────────┴───────────┘

Copy link
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very nice 👌 I kicked off some benchmarks

I don't think they show super meaningful improvements, but it doesn't hurt, so 👍 from me

Thanks @Dandandan

@Dandandan
Copy link
Contributor Author

Hm interesting.

I realize now push_batch_with_filter doesn't do any optimizations, so it makes sense it doesn't do much better than CoalesceBatches yet.

But there is some more potential, and we can remove the CoalesceBatches from the plan afterwards!

@Dandandan Dandandan added this pull request to the merge queue Nov 11, 2025
Merged via the queue into apache:main with commit 65bd13d Nov 11, 2025
32 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate physical-plan Changes to the physical-plan crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integrate Batch coalescing inside FilterExec

2 participants