Skip to content

coordinator: merge partial dynamic filters - #636

Open
jayshrivastava wants to merge 4 commits into
js/3-plan-dynamic-filtersfrom
js/4-merge-dynamic-filters
Open

coordinator: merge partial dynamic filters #636
jayshrivastava wants to merge 4 commits into
js/3-plan-dynamic-filtersfrom
js/4-merge-dynamic-filters

Conversation

@jayshrivastava

@jayshrivastava jayshrivastava commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Stack

This stack of PRs implements distributed dynamic filtering #528

  1. coordinator: display dynamic filters after execution #623
  2. feat: plan distributed dynamic filters #634
  3. feat: forward remote dynamic filter updates to coordinator #635
  4. coordinator: merge partial dynamic filters  #636 <- you are here
  5. coordinator: forward merged dynamic filters to consumers #637
  6. worker: apply merged dynamic filters during execution #639

Details

This PR adds machinery around merging dynamic filters in the DynamicFilterRegistry.

    register_task(stage1, task1)                                                                                              
    register_task(stage1, task2)      ─────────┐                                                                              
   register_task(stage1, task3)...             │          ┌───────────────────────┐      merge() when                         
         seal_stage(stage1)                    ├─────────▶│ DynamicFilterRegistry │───▶  - stage is sealed; and               
                                               │          └───────────────────────┘      - there's enough partial filter      
                                               │                                         updates                              
         add_partial(stage1, task1)   ─────────┘                                                                              
         add_partial(stage2, task1)                                                                                           
         add_partial(stage3, task1)                                                                                           
                                                                                                                              

The query coordinator is calling register_task on each SetPlanRequest. Concurrently, any running task can send a dynamic filter update to the registry via add_partial. The registry needs to detect when all the updates are present and merge() the partial dynamic filters. To help detect this, the coordinator is responsible for calling seal_stage once all the tasks have commenced so we know that no tasks will be added in the future.

The PR implements the above. In the next 2 PRs, we will actually forward the merged filters to consumers.

For now, we only care about completed dynamic filters. There's a follow up issue to look into incomplete dynamic filters: #665

@jayshrivastava
jayshrivastava changed the base branch from main to js/3-plan-dynamic-filters August 13, 2026 13:27
@jayshrivastava jayshrivastava changed the title Js/4 merge dynamic filters merge dynamic filters in Aug 13, 2026
@jayshrivastava jayshrivastava changed the title merge dynamic filters in coordinator: merge partial dynamic filters Aug 13, 2026
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 2cc815b to 7f6de9a Compare August 13, 2026 19:27
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch 2 times, most recently from 400db1c to ad3ea34 Compare August 17, 2026 19:05
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch 2 times, most recently from b798e6c to 876ac94 Compare August 18, 2026 18:15
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 876ac94 to d83e14d Compare August 18, 2026 18:52
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from d83e14d to 593f037 Compare August 21, 2026 16:28
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 593f037 to 1548203 Compare August 21, 2026 16:48
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 1548203 to 21b4e38 Compare August 21, 2026 20:58
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 21b4e38 to 4e36655 Compare August 22, 2026 15:00
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 4e36655 to 441ba38 Compare August 23, 2026 00:21
@jayshrivastava
jayshrivastava force-pushed the js/4-merge-dynamic-filters branch from 441ba38 to 2314a62 Compare August 23, 2026 15:27
Retain every latest producer report, including incomplete updates, and count each task completion exactly once. Most producers wait until all planned tasks report completion; collect-left hash joins narrowly use the first complete replica.

    task 0: incomplete F0 ---> retained
    task 0: complete F0 -----+
    task 1: complete F1 -----+--> flat protobuf OR --> merged F

    collect-left task 0: complete F0 --> merged F immediately
@jayshrivastava
jayshrivastava marked this pull request as ready for review August 23, 2026 19:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant