Test WideCombiner with 128 states from the beginning #16207

lll-phill-lll · 2025-03-25T12:04:49Z

No description provided.

lll-phill-lll · 2025-03-25T12:27:01Z

tpch1000 q4. spilling is enabled.

Tested on commit: 1e34c2c

1 state:

ydb -v workload tpch -p dt64/column/tpch/s1000 run --include 4 --iterations 10
https://paste.yandex-team.ru/ca224f21-3c33-460b-9ee7-6d6f2c4efe27

Query04:                                                                                                                                                           [410/6261]
        iteration 0:    ok      124.260084s seconds
        iteration 1:    ok      117.801391s seconds
        iteration 2:    ok      131.110469s seconds
        iteration 3:    ok      127.082814s seconds
        iteration 4:    ok      126.984910s seconds
        iteration 5:    ok      124.292087s seconds
        iteration 6:    ok      122.114472s seconds
        iteration 7:    ok      123.112514s seconds
        iteration 8:    ok      121.559919s seconds
        iteration 9:    ok      122.420739s seconds

Results for 10 iterations
┌─────────┬──────────┬─────────┬─────────┬─────────┬─────────┬───────────┬─────────┬─────────┬─────────┬─────────┬──────────────┬────────────┬────────────┐
│ Query # │ ColdTime │ Min     │ Max     │ Mean    │ Median  │ UnixBench │ Std     │ RttMin  │ RttMax  │ RttAvg  │ SuccessCount │ FailsCount │ DiffsCount │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Query04 │ 124.238  │ 117.781 │ 131.088 │ 124.052 │ 123.662 │ 121.839   │   3.469 │   0.012 │   0.026 │   0.022 │ 10           │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Sum     │ 124.238  │ 117.781 │ 131.088 │ 124.052 │ 123.662 │ 121.839   │   0.000 │   0.012 │   0.026 │   0.022 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Avg     │ 124.238  │ 117.781 │ 131.088 │ 124.052 │ 123.662 │ 121.839   │   0.000 │   0.012 │   0.026 │   0.022 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ GAvg    │ 124.238  │ 117.781 │ 131.088 │ 124.052 │ 123.662 │ 121.839   │   0.000 │   0.012 │   0.026 │   0.001 │ 1            │            │            │
└─────────┴──────────┴─────────┴─────────┴─────────┴─────────┴───────────┴─────────┴─────────┴─────────┴─────────┴──────────────┴────────────┴────────────┘

128 states

diff:

diff --git a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
index 85b4151d18d..2786c185e27 100644
--- a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
+++ b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
@@ -902,6 +902,7 @@ private:
     }

     bool IsSwitchToSpillingModeCondition() const {
+        return true;
         return !HasMemoryForProcessing() || TlsAllocState->GetMaximumLimitValueReached();
     }

ydb -v workload tpch -p dt64/column/tpch/s1000 run --include 4 --iterations 10
https://paste.yandex-team.ru/7d1b024d-bee2-440d-9f74-172528a8ad39

Query04:
        iteration 0:    ok      93.513983s seconds
        iteration 1:    ok      88.989485s seconds
        iteration 2:    ok      83.823538s seconds
        iteration 3:    ok      89.865635s seconds
        iteration 4:    ok      82.162986s seconds
        iteration 5:    ok      91.744036s seconds
        iteration 6:    ok      90.849011s seconds
        iteration 7:    ok      92.739788s seconds
        iteration 8:    ok      75.820480s seconds
        iteration 9:    ok      78.463166s seconds

Results for 10 iterations

┌────────┬──────────┬───────┬───────┬───────┬───────┬──────────┬───────┬────────┬────────┬────────┬────────┬───────┬───────┐
│Query # │Cold Time │Min    │Max    │Mean   │Median │Unix Bench│Std    │Rtt Min │Rtt Max │Rtt Avg │Success │Fails  │Diffs  │
├────────┼──────────┼───────┼───────┼───────┼───────┼──────────┼───────┼────────┼────────┼────────┼────────┼───────┼───────┤
│Qry04   │93.491    │75.795 │93.491 │86.772 │89.402 │83.005    │5.965  │0.022   │0.028   │0.024   │10      │       │       │
├────────┼──────────┼───────┼───────┼───────┼───────┼──────────┼───────┼────────┼────────┼────────┼────────┼───────┼───────┤
│Sum     │93.491    │75.795 │93.491 │86.772 │89.402 │83.005    │0.000  │0.022   │0.028   │0.024   │1       │       │       │
├────────┼──────────┼───────┼───────┼───────┼───────┼──────────┼───────┼────────┼────────┼────────┼────────┼───────┼───────┤
│Avg     │93.491    │75.795 │93.491 │86.772 │89.402 │83.005    │0.000  │0.022   │0.028   │0.024   │1       │       │       │
├────────┼──────────┼───────┼───────┼───────┼───────┼──────────┼───────┼────────┼────────┼────────┼────────┼───────┼───────┤
│GAvg    │93.491    │75.795 │93.491 │86.772 │89.402 │83.005    │0.000  │0.022   │0.028   │0.001   │1       │       │       │
└────────┴──────────┴───────┴───────┴───────┴───────┴──────────┴───────┴────────┴────────┴────────┴────────┴───────┴───────┘

Results

Immediate state splitting reduces computation time from 130 to 80 secs

lll-phill-lll · 2025-03-25T13:05:17Z

tpch1000 q4. spilling is disabled.

Tested on commit: 1e34c2c

1 state:

diff:

diff --git a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
index 85b4151d18d..41ab7f73375 100644
--- a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
+++ b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
@@ -32,6 +32,7 @@ extern TStatKey Combine_MaxRowsCount;
 namespace {

 bool HasMemoryForProcessing() {
+    return true;
     return !TlsAllocState->IsMemoryYellowZoneEnabled();
 }

@@ -902,6 +903,7 @@ private:
     }

     bool IsSwitchToSpillingModeCondition() const {
+        return false;
         return !HasMemoryForProcessing() || TlsAllocState->GetMaximumLimitValueReached();
     }

(END)

ydb -v workload tpch -p dt64/column/tpch/s1000 run --include 4 --iterations 10
https://paste.yandex-team.ru/d3215710-3299-49ec-a179-17fb35a5e1d3

Query04:
        iteration 0:    ok      45.877628s seconds
        iteration 1:    ok      50.604146s seconds
        iteration 2:    ok      47.365336s seconds
        iteration 3:    ok      47.729085s seconds
        iteration 4:    ok      46.227401s seconds
        iteration 5:    ok      46.581746s seconds
        iteration 6:    ok      47.886843s seconds
        iteration 7:    ok      46.594947s seconds
        iteration 8:    ok      47.006608s seconds
        iteration 9:    ok      47.377643s seconds

Results for 10 iterations
┌─────────┬──────────┬─────────┬─────────┬─────────┬─────────┬───────────┬─────────┬─────────┬─────────┬─────────┬──────────────┬────────────┬────────────┐
│ Query # │ ColdTime │ Min     │ Max     │ Mean    │ Median  │ UnixBench │ Std     │ RttMin  │ RttMax  │ RttAvg  │ SuccessCount │ FailsCount │ DiffsCount │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Query04 │  45.844  │  45.844 │  50.579 │  47.302 │  47.159 │  46.582   │   1.254 │   0.013 │   0.034 │   0.022 │ 10           │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Sum     │  45.844  │  45.844 │  50.579 │  47.302 │  47.159 │  46.582   │   0.000 │   0.013 │   0.034 │   0.022 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Avg     │  45.844  │  45.844 │  50.579 │  47.302 │  47.159 │  46.582   │   0.000 │   0.013 │   0.034 │   0.022 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ GAvg    │  45.844  │  45.844 │  50.579 │  47.302 │  47.159 │  46.582   │   0.000 │   0.013 │   0.034 │   0.001 │ 1            │            │            │
└─────────┴──────────┴─────────┴─────────┴─────────┴─────────┴───────────┴─────────┴─────────┴─────────┴─────────┴──────────────┴────────────┴────────────┘

128 states:

diff:

diff --git a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
index 85b4151d18d..e9746c77f41 100644
--- a/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
+++ b/yql/essentials/minikql/comp_nodes/mkql_wide_combine.cpp
@@ -32,6 +32,7 @@ extern TStatKey Combine_MaxRowsCount;
 namespace {

 bool HasMemoryForProcessing() {
+    return true;
     return !TlsAllocState->IsMemoryYellowZoneEnabled();
 }

@@ -902,6 +903,7 @@ private:
     }

     bool IsSwitchToSpillingModeCondition() const {
+        return true;
         return !HasMemoryForProcessing() || TlsAllocState->GetMaximumLimitValueReached();
     }

(END)

ydb -v workload tpch -p dt64/column/tpch/s1000 run --include 4 --iterations 10
https://paste.yandex-team.ru/3840e0ec-ee8a-48d1-ab9e-37340f55a01b

Query04:
        iteration 0:    ok      95.106456s seconds
        iteration 1:    ok      94.388362s seconds
        iteration 2:    ok      92.871570s seconds
        iteration 3:    ok      92.709556s seconds
        iteration 4:    ok      94.362293s seconds
        iteration 5:    ok      91.059113s seconds
        iteration 6:    ok      94.594771s seconds
        iteration 7:    ok      94.904465s seconds
        iteration 8:    ok      92.775566s seconds
        iteration 9:    ok      92.927785s seconds

Results for 10 iterations
┌─────────┬──────────┬─────────┬─────────┬─────────┬─────────┬───────────┬─────────┬─────────┬─────────┬─────────┬──────────────┬────────────┬────────────┐
│ Query # │ ColdTime │ Min     │ Max     │ Mean    │ Median  │ UnixBench │ Std     │ RttMin  │ RttMax  │ RttAvg  │ SuccessCount │ FailsCount │ DiffsCount │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Query04 │  95.082  │  91.039 │  95.082 │  93.544 │  93.618 │  92.755   │   1.225 │   0.021 │   0.035 │   0.025 │ 10           │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Sum     │  95.082  │  91.039 │  95.082 │  93.544 │  93.618 │  92.755   │   0.000 │   0.021 │   0.035 │   0.025 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ Avg     │  95.082  │  91.039 │  95.082 │  93.544 │  93.618 │  92.755   │   0.000 │   0.021 │   0.035 │   0.025 │ 1            │            │            │
├─────────┼──────────┼─────────┼─────────┼─────────┼─────────┼───────────┼─────────┼─────────┼─────────┼─────────┼──────────────┼────────────┼────────────┤
│ GAvg    │  95.082  │  91.039 │  95.082 │  93.544 │  93.618 │  92.755   │   0.000 │   0.021 │   0.035 │   0.001 │ 1            │            │            │
└─────────┴──────────┴─────────┴─────────┴─────────┴─────────┴───────────┴─────────┴─────────┴─────────┴─────────┴──────────────┴────────────┴────────────┘

lll-phill-lll · 2025-03-25T13:28:42Z

Time comparison:

+-------------+---------+------------+
|     ###     | 1 state | 128 states |
+-------------+---------+------------+
| Spilling    |     124 |         86 |
| No spilling |      47 |         93 |
+-------------+---------+------------+

lll-phill-lll · 2025-03-31T11:03:24Z

Decided to leave the current algorithm as is it

lll-phill-lll self-assigned this Mar 25, 2025

lll-phill-lll added the area/runtime YDB runtime issues label Mar 25, 2025

lll-phill-lll closed this as completed Mar 31, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test WideCombiner with 128 states from the beginning #16207

Test WideCombiner with 128 states from the beginning #16207

lll-phill-lll commented Mar 25, 2025

lll-phill-lll commented Mar 25, 2025 •

edited

Loading

lll-phill-lll commented Mar 25, 2025 •

edited

Loading

lll-phill-lll commented Mar 25, 2025 •

edited

Loading

lll-phill-lll commented Mar 31, 2025

Test WideCombiner with 128 states from the beginning #16207

Test WideCombiner with 128 states from the beginning #16207

Comments

lll-phill-lll commented Mar 25, 2025

lll-phill-lll commented Mar 25, 2025 • edited Loading

tpch1000 q4. spilling is enabled.

1 state:

128 states

Results

lll-phill-lll commented Mar 25, 2025 • edited Loading

tpch1000 q4. spilling is disabled.

1 state:

128 states:

lll-phill-lll commented Mar 25, 2025 • edited Loading

lll-phill-lll commented Mar 31, 2025

lll-phill-lll commented Mar 25, 2025 •

edited

Loading

lll-phill-lll commented Mar 25, 2025 •

edited

Loading

lll-phill-lll commented Mar 25, 2025 •

edited

Loading