Prefetch list elements from block boundaries - #9052
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (10)
Powered by Polar Signals Cloud |
Benchmarks: FineWeb NVMe 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.999x ➖, 0↑ 0↓)
datafusion / parquet (0.981x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.021x ➖, 0↑ 1↓)
duckdb / parquet (0.984x ➖, 0↑ 0↓)
File Size Changes (2 files changed, -46.3% overall, 0↑ 2↓)
Totals:
|
Benchmarks: PolarSignals Profiling 📖Vortex (geomean): 1.043x ➖ How to read Verdict and Engines
datafusion / vortex-file-compressed (1.043x ➖, 0↑ 2↓)
File Size Changes (1 files changed, -8.2% overall, 0↑ 1↓)
Totals:
|
Benchmarks: TPC-H SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.024x ➖, 0↑ 0↓)
datafusion / parquet (1.020x ➖, 1↑ 1↓)
duckdb / vortex-file-compressed (1.005x ➖, 0↑ 0↓)
duckdb / parquet (0.998x ➖, 0↑ 0↓)
File Size Changes (9 files changed, -43.9% overall, 0↑ 9↓)
Totals:
|
Benchmarks: TPC-H SF=10 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.010x ➖, 0↑ 0↓)
datafusion / parquet (1.010x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.014x ➖, 0↑ 0↓)
duckdb / parquet (0.988x ➖, 2↑ 0↓)
File Size Changes (9 files changed, -44.0% overall, 0↑ 9↓)
Totals:
|
Benchmarks: FineWeb S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.045x ➖, 0↑ 0↓)
datafusion / parquet (1.041x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed (1.072x ➖, 0↑ 0↓)
duckdb / parquet (1.024x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=1 on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.013x ➖, 0↑ 2↓)
datafusion / parquet (1.017x ➖, 0↑ 5↓)
duckdb / vortex-file-compressed (1.008x ➖, 2↑ 0↓)
duckdb / parquet (0.999x ➖, 1↑ 1↓)
File Size Changes (25 files changed, -43.5% overall, 0↑ 25↓)
Totals:
|
Benchmarks: Clickbench Sorted on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.085x ➖, 0↑ 3↓)
datafusion / parquet (1.030x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.028x ➖, 1↑ 3↓)
duckdb / parquet (1.017x ➖, 0↑ 0↓)
File Size Changes (201 files changed, -42.8% overall, 52↑ 149↓)
Totals:
|
Benchmarks: Statistical and Population Genetics 📖Verdict: Likely improvement (high confidence) How to read Verdict and Engines
duckdb / vortex-file-compressed (0.746x ✅, 9↑ 0↓)
duckdb / parquet (0.994x ➖, 0↑ 0↓)
File Size Changes (3 files changed, -42.8% overall, 0↑ 3↓)
Totals:
|
Benchmarks: TPC-H SF=1 on S3 📖Verdict: No clear signal (environment too noisy confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (1.046x ➖, 0↑ 0↓)
datafusion / parquet (1.033x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed (1.060x ➖, 0↑ 0↓)
duckdb / parquet (1.051x ➖, 0↑ 1↓)
|
Benchmarks: Clickbench on NVME 📖Verdict: No clear signal (low confidence) How to read Verdict and Engines
datafusion / vortex-file-compressed (0.949x ➖, 7↑ 0↓)
datafusion / parquet (0.994x ➖, 3↑ 1↓)
duckdb / vortex-file-compressed (1.015x ➖, 1↑ 2↓)
duckdb / parquet (1.010x ➖, 0↑ 1↓)
File Size Changes (101 files changed, -39.2% overall, 0↑ 101↓)
Totals:
|
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
Signed-off-by: Matt Katz <mhkatz97@gmail.com>
2f1b871 to
5a391cf
Compare
Summary
Motivation
Nested structural-list reads currently have to resolve offsets before they can issue the corresponding element read. Deeply nested lists have several steps of sequential io to fetch offsets and then elements. Coarse paired boundaries provide a safe element envelope so broad reads can overlap these operations without flattening the list layout.