Skip to content

Add parquet V1 rep/def prefix decompress path#769

Open
liuneng1994 wants to merge 1 commit into
bytedance:mainfrom
liuneng1994:parquet-v1-repdef-prefix-decode
Open

Add parquet V1 rep/def prefix decompress path#769
liuneng1994 wants to merge 1 commit into
bytedance:mainfrom
liuneng1994:parquet-v1-repdef-prefix-decode

Conversation

@liuneng1994

@liuneng1994 liuneng1994 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

What problem does this PR solve?

Parquet Data Page V1 stores repetition and definition levels as a prefix of the page body. When Bolt preloads rep/def information for nested columns, the existing path decompresses the whole page even though only the rep/def prefix is needed at that stage.

This is wasteful for nested columns with large value payloads after the prefix. The overhead is concentrated in the rep/def preload path, before leaf values are materialized.

Issue Number: N/A

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance improvement (optimization)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring (no logic changes)
  • Build/CI or Infrastructure changes
  • Documentation only

Description

This PR adds a narrow fast path in PageReader for Parquet V1 rep/def-only reads.

The optimization is intentionally scoped to row == kRepDefOnly, which is the path used while preloading rep/def data for non-top-level nested Parquet columns. In that case, PageReader now attempts to decompress only enough of the page body to parse the V1 repetition/definition prefix.

If the prefix cannot be decoded from the partial output, the reader falls back to the existing full-page decompression path. The normal Page V1 full-decompress path remains unchanged.

Supported fast-path cases in this PR:

  • Parquet Data Page V1
  • rep/def-only page reads (row == kRepDefOnly)
  • ZSTD compressed pages
  • UNCOMPRESSED pages

This PR intentionally excludes unrelated reader refactors, level-conversion changes, and ZSTD context reuse changes.

Performance Impact

  • No Impact: This change does not affect the critical path (e.g., build system, doc, error handling).
  • Positive Impact: I have run benchmarks.
Click to view Benchmark Results
Benchmark: bolt_dwio_parquet_page_reader_preload_benchmark
Workload : synthetic multi-page Parquet V1 + ZSTD pages
Scenario : direct comparison of rep/def preload paths only
          prefix_off = full-page decompress before extracting rep/def
          prefix_on  = decode rep/def prefix without decompressing value payload

prefix_off: 51.04ms
prefix_on :  1.08ms
Speedup   : ~47x

This benchmark is intentionally low-level and isolates the exact hotspot this PR optimizes: preloading rep/def data from Parquet V1 pages when the value payload is large but not needed yet.

The previously tried end-to-end nested read benchmark did not reliably expose the difference because broader reader and materialization costs dominated the total runtime. The new microbenchmark measures the preload path directly, which is the intended optimization target.

  • Negative Impact: Explained below (e.g., trade-off for correctness).

Release Note

Release Note:
- Optimized Parquet V1 nested-column rep/def preloading by avoiding full-page decompression when only the rep/def prefix is needed.

Checklist (For Author)

  • I have added/updated unit tests (ctest).
  • I have verified the code with local build (Release/Debug).
  • I have run clang-format / linters.
  • (Optional) I have run Sanitizers (ASAN/TSAN) locally for complex C++ changes.
  • No need to test or manual test.

Validation commands:

cmake --build --preset conan-release --target bolt_dwio_parquet_reader_test bolt_dwio_parquet_page_reader_preload_benchmark --parallel 32
_build/Release/bolt/dwio/parquet/tests/reader/bolt_dwio_parquet_reader_test --gtest_filter=ParquetPageReaderTest.zstdDataPageV1RepDefPrefix
_build/Release/bolt/dwio/parquet/tests/reader/bolt_dwio_parquet_page_reader_preload_benchmark --bm_min_iters=5

Breaking Changes

  • No
  • Yes (Description: ...)

@CLAassistant

CLAassistant commented Jul 22, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@liuneng1994
liuneng1994 force-pushed the parquet-v1-repdef-prefix-decode branch 2 times, most recently from 5037aae to b418bb9 Compare July 23, 2026 08:58
}
}

bool PageReader::tryPrepareDataPageV1RepDefOnly(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tryPrepareDataPageV1RepDefOnly() currently mixes several responsibilities in one large function: detecting whether the rep/def prefix is complete, partially decompressing standard ZSTD frames, parsing the
custom chunked ZSTD layout, setting up rep/def decoders, appending raw rep/def data, and updating PageReader state/statistics. This makes the change harder to review and maintain, especially because the
Parquet page-layout logic and the codec-specific decompression logic are at different abstraction levels.

There is also a fair amount of duplicated logic from the existing prepareDataPageV1() path, especially around parsing repetition/definition level lengths, initializing decoders, updating totalRefDefBytes_,
and copying raw rep/def bytes into preloadedRepDefs_. It would be better to extract small shared helpers for the V1 rep/def prefix handling, so both the full-decompression fallback and the prefix fast path
use the same parsing/copying code.

The partial decompression part likely needs new code because the existing decompressData() API always materializes the full page, but that codec-specific logic could be isolated from PageReader more
cleanly. For example, PageReader could ask for “enough uncompressed bytes to satisfy this prefix predicate”, while the ZSTD frame/chunk parsing stays in a decompression helper.

Comment thread bolt/dwio/parquet/reader/PageReader.h Outdated
/// continuous stream accessible via readWithVisitor().
class PageReader {
public:
BOLT_FRIEND_TEST(ParquetPageReaderTest, zstdDataPageV1RepDefPrefix);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having the benchmark reach into PageReader internals suggests the test surface is too white-box. If the prefix-decompression or rep/def-prefix parsing were factored into smaller helpers, those could be tested or benchmarked more directly without exposing additional private access from the production header.

@liuneng1994
liuneng1994 force-pushed the parquet-v1-repdef-prefix-decode branch 3 times, most recently from 6ed53b0 to 8c85aad Compare July 24, 2026 07:33
Optimize DataPage V1 rep/def preload by decompressing only the rep/def prefix when possible, and share the V1 rep/def parsing, raw-copy, and decoder setup logic with the normal page prepare path.

Add a focused page reader benchmark and public-API test coverage for the prefix preload path without adding test-only hooks to production code.

Co-authored-by: TRAE CLI <noreply@bytedance.com>
@liuneng1994
liuneng1994 force-pushed the parquet-v1-repdef-prefix-decode branch from 8c85aad to efd9953 Compare July 24, 2026 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants