Part of #522.
Use statistics collected by runtime sampling for swapping JOIN orders during dynamic planning.
The plan mutation should likely happen right in this point:
|
|
|
// Propagate the final task_count inferred based on runtime statistics and compute cost. |
|
// Here is where leaf nodes are scaled up by ScaleUpLeafNodeHandler, and the |
|
// plan is finally left ready for distribution. |
|
input_stage.plan = nb_ctx |
At that point, all the NetworkBoundary nodes below have already collected runtime stats, so we have a very good intuition about the incoming data.
It would be cool to rely on upstream rules for join selection, but I'm not sure if it's possible.
For more info about the background of this issue, see #522.
Part of #522.
Use statistics collected by runtime sampling for swapping JOIN orders during dynamic planning.
The plan mutation should likely happen right in this point:
datafusion-distributed/src/coordinator/prepare_dynamic_plan.rs
Lines 71 to 75 in 81b887e
At that point, all the
NetworkBoundarynodes below have already collected runtime stats, so we have a very good intuition about the incoming data.It would be cool to rely on upstream rules for join selection, but I'm not sure if it's possible.
For more info about the background of this issue, see #522.