Add parquet V1 rep/def prefix decompress path#769
Conversation
5037aae to
b418bb9
Compare
| } | ||
| } | ||
|
|
||
| bool PageReader::tryPrepareDataPageV1RepDefOnly( |
There was a problem hiding this comment.
tryPrepareDataPageV1RepDefOnly() currently mixes several responsibilities in one large function: detecting whether the rep/def prefix is complete, partially decompressing standard ZSTD frames, parsing the
custom chunked ZSTD layout, setting up rep/def decoders, appending raw rep/def data, and updating PageReader state/statistics. This makes the change harder to review and maintain, especially because the
Parquet page-layout logic and the codec-specific decompression logic are at different abstraction levels.
There is also a fair amount of duplicated logic from the existing prepareDataPageV1() path, especially around parsing repetition/definition level lengths, initializing decoders, updating totalRefDefBytes_,
and copying raw rep/def bytes into preloadedRepDefs_. It would be better to extract small shared helpers for the V1 rep/def prefix handling, so both the full-decompression fallback and the prefix fast path
use the same parsing/copying code.
The partial decompression part likely needs new code because the existing decompressData() API always materializes the full page, but that codec-specific logic could be isolated from PageReader more
cleanly. For example, PageReader could ask for “enough uncompressed bytes to satisfy this prefix predicate”, while the ZSTD frame/chunk parsing stays in a decompression helper.
| /// continuous stream accessible via readWithVisitor(). | ||
| class PageReader { | ||
| public: | ||
| BOLT_FRIEND_TEST(ParquetPageReaderTest, zstdDataPageV1RepDefPrefix); |
There was a problem hiding this comment.
Having the benchmark reach into PageReader internals suggests the test surface is too white-box. If the prefix-decompression or rep/def-prefix parsing were factored into smaller helpers, those could be tested or benchmarked more directly without exposing additional private access from the production header.
6ed53b0 to
8c85aad
Compare
Optimize DataPage V1 rep/def preload by decompressing only the rep/def prefix when possible, and share the V1 rep/def parsing, raw-copy, and decoder setup logic with the normal page prepare path. Add a focused page reader benchmark and public-API test coverage for the prefix preload path without adding test-only hooks to production code. Co-authored-by: TRAE CLI <noreply@bytedance.com>
8c85aad to
efd9953
Compare
What problem does this PR solve?
Parquet Data Page V1 stores repetition and definition levels as a prefix of the page body. When Bolt preloads rep/def information for nested columns, the existing path decompresses the whole page even though only the rep/def prefix is needed at that stage.
This is wasteful for nested columns with large value payloads after the prefix. The overhead is concentrated in the rep/def preload path, before leaf values are materialized.
Issue Number: N/A
Type of Change
Description
This PR adds a narrow fast path in
PageReaderfor Parquet V1 rep/def-only reads.The optimization is intentionally scoped to
row == kRepDefOnly, which is the path used while preloading rep/def data for non-top-level nested Parquet columns. In that case,PageReadernow attempts to decompress only enough of the page body to parse the V1 repetition/definition prefix.If the prefix cannot be decoded from the partial output, the reader falls back to the existing full-page decompression path. The normal Page V1 full-decompress path remains unchanged.
Supported fast-path cases in this PR:
row == kRepDefOnly)This PR intentionally excludes unrelated reader refactors, level-conversion changes, and ZSTD context reuse changes.
Performance Impact
Click to view Benchmark Results
This benchmark is intentionally low-level and isolates the exact hotspot this PR optimizes: preloading rep/def data from Parquet V1 pages when the value payload is large but not needed yet.
The previously tried end-to-end nested read benchmark did not reliably expose the difference because broader reader and materialization costs dominated the total runtime. The new microbenchmark measures the preload path directly, which is the intended optimization target.
Release Note
Checklist (For Author)
Validation commands:
Breaking Changes