Redesign join API and internals#78
Draft
gene-bordegaray wants to merge 2 commits into
Draft
Conversation
2295418 to
4bffe3a
Compare
4bffe3a to
8478b4a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR is stacked on top of the root execution/readiness API redesign in #77. It reshapes the libcudf-rs join API and internals around smaller, explicit join paths.
It replaces the monolithic join implementation with focused modules for common selection logic, equi joins, hash joins, join indices, masks, and output gathering.
Stream API details
The reusable hash join can be converted into a streaming join with
into_streaming_join(). Streaming probes record matched build indices as each probe batch runs, andunmatched_build_rows(...)finalizes left/full-style output by gathering build rows that were never matched.Filtered streaming probes use the same builder-style API as regular probes:
filter(predicate), optionalcondition_tables(...), optional payload overrides, and selected build/probe outputs. The filtered path computes equality candidates first, applies the AST predicate, records only surviving build matches, and appends unmatched probe rows for left probes.The stream readiness model keeps hash join state, filtered build state, and join index vectors alive until dependent GPU work is ready.
Internal structure
src/join.rsintosrc/join/{common,equi,hash,indices,masks,output}.rs.CuDFErrorpaths.Validation
git diff --cached --checkcargo fmt --checkcargo check --workspacecargo clippy -p libcudf-rs -- -D warningscargo test -p libcudf-rs join:: -- --nocapturecargo test -p libcudf-datafusion physical::hash_join -- --test-threads=1 --nocapturecargo test -p libcudf-datafusion expr::ast -- --test-threads=1 --nocapturecargo doc -p libcudf-rs --no-deps