coordinator: merge partial dynamic filters - #636
Open
jayshrivastava wants to merge 4 commits into
Open
Conversation
This was referenced Aug 13, 2026
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 13, 2026 19:27
2cc815b to
7f6de9a
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
2 times, most recently
from
August 17, 2026 19:05
400db1c to
ad3ea34
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
2 times, most recently
from
August 18, 2026 18:15
b798e6c to
876ac94
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 18, 2026 18:52
876ac94 to
d83e14d
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 21, 2026 16:28
d83e14d to
593f037
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 21, 2026 16:48
593f037 to
1548203
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 21, 2026 20:58
1548203 to
21b4e38
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 22, 2026 15:00
21b4e38 to
4e36655
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 23, 2026 00:21
4e36655 to
441ba38
Compare
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 23, 2026 15:27
441ba38 to
2314a62
Compare
Retain every latest producer report, including incomplete updates, and count each task completion exactly once. Most producers wait until all planned tasks report completion; collect-left hash joins narrowly use the first complete replica.
task 0: incomplete F0 ---> retained
task 0: complete F0 -----+
task 1: complete F1 -----+--> flat protobuf OR --> merged F
collect-left task 0: complete F0 --> merged F immediately
jayshrivastava
force-pushed
the
js/4-merge-dynamic-filters
branch
from
August 23, 2026 15:35
2314a62 to
af77f78
Compare
jayshrivastava
marked this pull request as ready for review
August 23, 2026 19:28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack
This stack of PRs implements distributed dynamic filtering #528
Details
This PR adds machinery around merging dynamic filters in the
DynamicFilterRegistry.The query coordinator is calling
register_taskon eachSetPlanRequest. Concurrently, any running task can send a dynamic filter update to the registry viaadd_partial. The registry needs to detect when all the updates are present andmerge()the partial dynamic filters. To help detect this, the coordinator is responsible for callingseal_stageonce all the tasks have commenced so we know that no tasks will be added in the future.The PR implements the above. In the next 2 PRs, we will actually forward the merged filters to consumers.
For now, we only care about completed dynamic filters. There's a follow up issue to look into incomplete dynamic filters: #665