duckdb: Parallelise on Partition, not ArrayRef - #9193
Conversation
Merging this PR will improve performance by 21.42%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Verdict: Likely regression (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.048x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (4.809x ❌, 0↑ 9↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -46.3% overall, 0↑ 2↓)
Totals:
|
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.054x ➖ datafusion / vortex-file-compressed / ns (1.054x ➖, 0↑ 2↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: Likely regression (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.038x ➖, 0↑ 2↓)
datafusion / parquet / ns (1.006x ➖, 1↑ 2↓)
duckdb / vortex-file-compressed / ns (3.210x ❌, 0↑ 22↓)
duckdb / parquet / ns (1.001x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -43.9% overall, 0↑ 9↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: Likely regression (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.009x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (8.943x ❌, 0↑ 22↓)
duckdb / parquet / ns (1.012x ➖, 0↑ 1↓)
File Size Changes (9 files changed, -44.0% overall, 0↑ 9↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.127x ❌, 0↑ 1↓)
datafusion / parquet / ns (0.980x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed / ns (1.230x ❌, 2↑ 7↓)
duckdb / parquet / ns (0.997x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -42.8% overall, 50↑ 151↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.001x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (0.993x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.041x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.163x ➖, 0↑ 6↓)
duckdb / vortex-compact / ns (1.377x ❌, 0↑ 10↓)
duckdb / parquet / ns (1.009x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: Likely regression (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.022x ➖, 0↑ 3↓)
datafusion / parquet / ns (0.996x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (2.331x ❌, 1↑ 93↓)
duckdb / parquet / ns (1.008x ➖, 5↑ 4↓)
File Size Changes (25 files changed, -43.5% overall, 0↑ 25↓)
Totals:
|
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.018x ➖, 1↑ 2↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.146x ❌, 6↑ 23↓)
duckdb / parquet / ns (1.006x ➖, 0↑ 0↓)
File Size Changes (101 files changed, -39.2% overall, 0↑ 101↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Verdict: Likely regression (high confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed / ns (3.975x ❌, 1↑ 9↓)
duckdb / parquet / ns (1.007x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -32.3% overall, 0↑ 2↓)
Totals:
|
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.941x ➖, 0↑ 0↓)
datafusion / vortex-compact / ns (1.137x ➖, 0↑ 1↓)
datafusion / parquet / ns (1.054x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (1.249x ➖, 0↑ 3↓)
duckdb / vortex-compact / ns (3.731x ❌, 0↑ 7↓)
duckdb / parquet / ns (1.271x ➖, 0↑ 1↓)
|
Benchmarks: Appian on NVME 📖Verdict: Likely regression (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.012x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.004x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (6.250x ❌, 0↑ 8↓)
duckdb / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / duckdb / ns (1.006x ➖, 0↑ 0↓)
File Size Changes (1 files changed, -0.0% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=10 on S3 📖Verdict: Likely regression (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.078x ➖, 1↑ 4↓)
datafusion / vortex-compact / ns (0.955x ➖, 1↑ 1↓)
datafusion / parquet / ns (1.014x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (2.034x ❌, 0↑ 18↓)
duckdb / vortex-compact / ns (3.055x ❌, 0↑ 20↓)
duckdb / parquet / ns (0.960x ➖, 0↑ 0↓)
|
Benchmarks: Vortex queries 📖Verdict: Likely regression (medium confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (5.372x ❌, 0↑ 3↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
TODO