Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[performance](load) fix broker load scan ranges for unsplittable files #43161

Merged
merged 4 commits into from
Nov 14, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented Nov 4, 2024

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

kaijchen commented Nov 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41457 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87a37bbf2e484d483d27d1e2a14d2d747fe370bb, data reload: false

------ Round 1 ----------------------------------
q1	17603	7460	7302	7302
q2	2047	176	166	166
q3	10718	1074	1165	1074
q4	10555	935	779	779
q5	7753	3061	3053	3053
q6	233	146	145	145
q7	1006	597	593	593
q8	9349	1932	2043	1932
q9	6612	6411	6471	6411
q10	7014	2454	2459	2454
q11	465	258	254	254
q12	413	215	218	215
q13	17773	3015	3012	3012
q14	238	209	215	209
q15	577	526	511	511
q16	652	570	591	570
q17	963	543	548	543
q18	7251	6795	6691	6691
q19	1354	1041	1107	1041
q20	462	183	187	183
q21	3974	3298	3347	3298
q22	1122	1021	1031	1021
Total cold run time: 108134 ms
Total hot run time: 41457 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7319	7273	7310	7273
q2	321	227	234	227
q3	3014	2959	2975	2959
q4	2134	1836	1845	1836
q5	5723	5759	5796	5759
q6	227	140	138	138
q7	2277	1803	1775	1775
q8	3395	3523	3515	3515
q9	8954	8918	8780	8780
q10	3616	3542	3541	3541
q11	598	513	514	513
q12	826	626	630	626
q13	9927	3183	3196	3183
q14	324	286	290	286
q15	582	518	545	518
q16	682	649	649	649
q17	1868	1630	1633	1630
q18	8321	7824	7619	7619
q19	1705	1576	1551	1551
q20	2086	1889	1933	1889
q21	5498	5413	5359	5359
q22	1146	1052	1046	1046
Total cold run time: 70543 ms
Total hot run time: 60672 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196298 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87a37bbf2e484d483d27d1e2a14d2d747fe370bb, data reload: false

query1	1196	897	887	887
query2	6248	2087	2156	2087
query3	10792	3973	4070	3973
query4	67547	29492	23707	23707
query5	4967	452	429	429
query6	398	163	166	163
query7	5668	289	286	286
query8	300	222	223	222
query9	9444	2731	2731	2731
query10	447	263	244	244
query11	17492	15360	15872	15360
query12	153	109	109	109
query13	1550	456	427	427
query14	10631	6859	7169	6859
query15	223	204	187	187
query16	7063	460	481	460
query17	1021	562	555	555
query18	1810	309	294	294
query19	202	152	149	149
query20	113	110	110	110
query21	199	103	101	101
query22	4604	4443	4371	4371
query23	35243	34299	34002	34002
query24	5957	2844	2803	2803
query25	514	402	401	401
query26	650	158	157	157
query27	1689	294	297	294
query28	4065	2463	2396	2396
query29	692	439	434	434
query30	235	147	157	147
query31	973	778	791	778
query32	65	57	55	55
query33	450	271	278	271
query34	929	525	512	512
query35	864	751	746	746
query36	1062	970	943	943
query37	121	73	74	73
query38	4493	4241	4331	4241
query39	1448	1429	1448	1429
query40	196	100	99	99
query41	49	45	45	45
query42	109	100	99	99
query43	565	520	514	514
query44	1170	822	825	822
query45	180	166	166	166
query46	1139	718	709	709
query47	1973	1856	1875	1856
query48	452	311	321	311
query49	746	399	402	399
query50	829	403	416	403
query51	7274	7163	7126	7126
query52	99	92	86	86
query53	249	181	189	181
query54	524	420	386	386
query55	74	74	72	72
query56	256	219	236	219
query57	1278	1200	1146	1146
query58	211	203	201	201
query59	3278	3071	3039	3039
query60	265	250	251	250
query61	130	101	100	100
query62	793	676	683	676
query63	216	188	185	185
query64	1341	617	607	607
query65	3296	3227	3227	3227
query66	694	299	303	299
query67	15918	15808	15615	15615
query68	3595	564	583	564
query69	414	260	247	247
query70	1205	1140	1131	1131
query71	344	250	248	248
query72	6362	4045	3979	3979
query73	747	352	358	352
query74	10112	9036	9140	9036
query75	3375	2666	2710	2666
query76	1736	1143	1100	1100
query77	452	271	264	264
query78	10359	9447	9424	9424
query79	1519	598	584	584
query80	848	417	427	417
query81	522	245	243	243
query82	1277	114	111	111
query83	184	136	136	136
query84	275	71	69	69
query85	868	290	290	290
query86	336	290	308	290
query87	4800	4764	4631	4631
query88	3679	2192	2152	2152
query89	417	299	291	291
query90	1997	188	185	185
query91	133	104	99	99
query92	68	51	49	49
query93	1911	539	545	539
query94	791	296	289	289
query95	347	253	245	245
query96	626	282	282	282
query97	2912	2654	2689	2654
query98	212	190	192	190
query99	1578	1310	1321	1310
Total cold run time: 318823 ms
Total hot run time: 196298 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87a37bbf2e484d483d27d1e2a14d2d747fe370bb, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.04
query3	0.23	0.06	0.07
query4	1.64	0.11	0.10
query5	0.43	0.40	0.42
query6	1.15	0.66	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.56	0.56
query11	0.14	0.11	0.11
query12	0.12	0.11	0.11
query13	0.61	0.59	0.60
query14	2.75	2.73	2.74
query15	0.90	0.84	0.83
query16	0.38	0.37	0.40
query17	1.07	1.06	1.06
query18	0.19	0.20	0.20
query19	2.00	1.87	1.92
query20	0.01	0.01	0.01
query21	15.37	0.60	0.57
query22	2.66	2.73	1.19
query23	16.80	1.11	0.73
query24	2.58	1.28	2.47
query25	0.22	0.15	0.07
query26	0.62	0.14	0.13
query27	0.04	0.04	0.05
query28	9.57	1.11	1.07
query29	12.54	3.23	3.24
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.27	0.46	0.45
query33	2.98	2.96	3.08
query34	17.10	4.45	4.49
query35	4.50	4.48	4.50
query36	0.66	0.48	0.48
query37	0.09	0.06	0.05
query38	0.06	0.03	0.04
query39	0.03	0.02	0.03
query40	0.16	0.12	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.41 s
Total hot run time: 32.08 s

@kaijchen kaijchen marked this pull request as ready for review November 6, 2024 01:13
@kaijchen
Copy link
Contributor Author

kaijchen commented Nov 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41063 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f6a294fbb4fd126155c5079cbdc905c86e53b5b, data reload: false

------ Round 1 ----------------------------------
q1	17576	7441	7316	7316
q2	2040	157	149	149
q3	10635	1079	1136	1079
q4	10580	848	745	745
q5	7732	3041	3006	3006
q6	241	145	146	145
q7	994	603	599	599
q8	9356	1995	2018	1995
q9	6515	6441	6446	6441
q10	7102	2427	2393	2393
q11	467	264	256	256
q12	410	214	210	210
q13	17772	2985	3029	2985
q14	248	216	210	210
q15	583	548	536	536
q16	673	594	585	585
q17	966	478	684	478
q18	7326	6672	6588	6588
q19	1339	1040	980	980
q20	476	181	185	181
q21	3960	3195	3194	3194
q22	1072	1024	992	992
Total cold run time: 108063 ms
Total hot run time: 41063 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7361	7306	7359	7306
q2	326	233	221	221
q3	2948	2968	2995	2968
q4	2084	1855	1880	1855
q5	5783	5774	5865	5774
q6	226	138	140	138
q7	2275	1848	1812	1812
q8	3396	3461	3497	3461
q9	8969	8965	8915	8915
q10	3619	3600	3559	3559
q11	620	517	496	496
q12	833	648	648	648
q13	10505	3263	3173	3173
q14	316	275	288	275
q15	562	521	523	521
q16	696	651	635	635
q17	1837	1671	1666	1666
q18	8177	7920	7540	7540
q19	1732	1456	1558	1456
q20	2145	1881	1879	1879
q21	5669	5403	5548	5403
q22	1164	1071	1021	1021
Total cold run time: 71243 ms
Total hot run time: 60722 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196738 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f6a294fbb4fd126155c5079cbdc905c86e53b5b, data reload: false

query1	1184	939	928	928
query2	6233	2102	2043	2043
query3	10780	4130	3755	3755
query4	67931	28740	23704	23704
query5	4979	441	426	426
query6	402	181	182	181
query7	5643	293	304	293
query8	315	225	229	225
query9	9315	2630	2641	2630
query10	453	263	240	240
query11	17492	15354	15910	15354
query12	152	105	103	103
query13	1543	415	431	415
query14	10882	7720	7735	7720
query15	222	186	198	186
query16	7070	454	480	454
query17	1017	567	560	560
query18	1777	301	294	294
query19	204	153	167	153
query20	119	112	112	112
query21	201	102	99	99
query22	4561	4232	4292	4232
query23	34499	34114	34297	34114
query24	6061	2779	2810	2779
query25	508	398	402	398
query26	646	160	161	160
query27	1704	284	288	284
query28	4124	2421	2429	2421
query29	687	429	428	428
query30	238	151	156	151
query31	994	809	781	781
query32	68	56	58	56
query33	439	269	269	269
query34	919	530	521	521
query35	893	731	749	731
query36	1045	965	983	965
query37	122	78	74	74
query38	4437	4352	4275	4275
query39	1482	1447	1461	1447
query40	204	102	100	100
query41	47	44	45	44
query42	108	100	95	95
query43	527	498	501	498
query44	1162	822	837	822
query45	183	169	169	169
query46	1129	726	721	721
query47	1974	1816	1869	1816
query48	423	311	317	311
query49	755	420	407	407
query50	809	401	394	394
query51	7279	7187	7117	7117
query52	100	91	85	85
query53	265	182	186	182
query54	535	402	414	402
query55	74	73	77	73
query56	244	247	246	246
query57	1246	1164	1174	1164
query58	231	206	199	199
query59	3146	2946	3006	2946
query60	275	247	249	247
query61	110	114	111	111
query62	800	664	708	664
query63	225	194	191	191
query64	1353	659	623	623
query65	3341	3251	3263	3251
query66	695	301	307	301
query67	15808	15633	15531	15531
query68	3779	573	558	558
query69	412	260	263	260
query70	1122	1149	1154	1149
query71	328	258	256	256
query72	6448	4019	4001	4001
query73	765	353	360	353
query74	9991	8974	9133	8974
query75	3370	2672	2685	2672
query76	1738	1063	1063	1063
query77	471	294	266	266
query78	10322	9364	9452	9364
query79	1487	596	594	594
query80	880	418	429	418
query81	521	245	242	242
query82	1288	120	114	114
query83	219	141	143	141
query84	279	68	67	67
query85	881	306	296	296
query86	324	300	285	285
query87	4779	4773	4719	4719
query88	3645	2180	2157	2157
query89	412	297	291	291
query90	1990	188	184	184
query91	133	104	103	103
query92	60	46	47	46
query93	1914	540	534	534
query94	770	285	284	284
query95	344	252	244	244
query96	627	280	278	278
query97	2915	2725	2664	2664
query98	210	196	199	196
query99	1591	1294	1308	1294
Total cold run time: 318404 ms
Total hot run time: 196738 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f6a294fbb4fd126155c5079cbdc905c86e53b5b, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.22	0.07	0.07
query4	1.64	0.11	0.11
query5	0.42	0.42	0.40
query6	1.14	0.65	0.64
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.49
query10	0.55	0.56	0.56
query11	0.14	0.11	0.12
query12	0.14	0.11	0.10
query13	0.61	0.59	0.59
query14	2.72	2.82	2.76
query15	0.89	0.82	0.82
query16	0.39	0.40	0.36
query17	1.07	1.07	1.05
query18	0.19	0.19	0.20
query19	1.91	1.79	1.95
query20	0.02	0.01	0.01
query21	15.35	0.58	0.58
query22	2.61	2.08	1.98
query23	16.91	0.93	0.93
query24	2.97	1.98	2.28
query25	0.24	0.08	0.07
query26	0.61	0.14	0.14
query27	0.05	0.05	0.04
query28	9.02	1.11	1.07
query29	12.58	3.22	3.20
query30	0.25	0.07	0.06
query31	2.87	0.37	0.39
query32	3.28	0.46	0.45
query33	2.99	3.01	3.06
query34	17.23	4.45	4.50
query35	4.51	4.48	4.47
query36	0.67	0.48	0.48
query37	0.08	0.06	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.38 s
Total hot run time: 33.73 s

@kaijchen
Copy link
Contributor Author

kaijchen commented Nov 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41277 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b735f7cc1fe1d9cc71a26b210986b6f5ae336377, data reload: false

------ Round 1 ----------------------------------
q1	17589	7372	7227	7227
q2	2047	185	184	184
q3	10579	1086	1209	1086
q4	10564	848	812	812
q5	7753	3055	3036	3036
q6	237	146	144	144
q7	1006	609	625	609
q8	9396	1928	2044	1928
q9	6566	6436	6442	6436
q10	7084	2408	2403	2403
q11	468	262	261	261
q12	413	210	204	204
q13	18021	3053	3018	3018
q14	235	209	204	204
q15	564	520	524	520
q16	649	571	597	571
q17	1010	624	628	624
q18	7180	6965	6650	6650
q19	1344	986	1072	986
q20	477	186	188	186
q21	3958	3262	3203	3203
q22	1096	992	985	985
Total cold run time: 108236 ms
Total hot run time: 41277 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7266	7209	7309	7209
q2	348	254	252	252
q3	2988	2924	2934	2924
q4	2065	1827	1787	1787
q5	5710	5776	5784	5776
q6	226	144	144	144
q7	2193	1821	1894	1821
q8	3368	3577	3442	3442
q9	8946	8980	8886	8886
q10	3587	3575	3534	3534
q11	607	530	493	493
q12	834	635	627	627
q13	9425	3209	3208	3208
q14	301	278	274	274
q15	602	558	550	550
q16	688	651	655	651
q17	1864	1633	1610	1610
q18	8272	7961	7486	7486
q19	1718	1638	1572	1572
q20	2099	1866	1865	1865
q21	5624	5469	5529	5469
q22	1126	1005	993	993
Total cold run time: 69857 ms
Total hot run time: 60573 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194481 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b735f7cc1fe1d9cc71a26b210986b6f5ae336377, data reload: false

query1	2305	2138	2261	2138
query2	6252	2163	2043	2043
query3	15016	11296	247	247
query4	33253	23939	23641	23641
query5	3480	461	444	444
query6	256	180	177	177
query7	3974	300	298	298
query8	292	228	228	228
query9	9612	2705	2699	2699
query10	478	249	262	249
query11	17754	15383	15391	15383
query12	158	109	103	103
query13	1608	428	407	407
query14	9441	7863	6928	6928
query15	256	180	187	180
query16	7828	456	483	456
query17	1578	578	582	578
query18	1794	675	641	641
query19	247	192	184	184
query20	123	119	112	112
query21	203	106	111	106
query22	4613	4176	4396	4176
query23	34866	34296	33918	33918
query24	11670	3259	3308	3259
query25	616	402	400	400
query26	1249	175	174	174
query27	2498	285	284	284
query28	7628	2453	2419	2419
query29	872	435	419	419
query30	396	321	308	308
query31	1011	807	815	807
query32	98	56	58	56
query33	773	287	284	284
query34	912	510	511	510
query35	874	791	731	731
query36	1100	948	946	946
query37	121	77	72	72
query38	4439	4348	4392	4348
query39	1438	1426	1429	1426
query40	269	100	103	100
query41	48	52	48	48
query42	113	96	103	96
query43	512	504	497	497
query44	1327	813	813	813
query45	186	170	172	170
query46	1142	720	701	701
query47	1916	1836	1848	1836
query48	423	318	330	318
query49	945	388	404	388
query50	818	394	396	394
query51	7325	7065	7154	7065
query52	99	89	86	86
query53	249	172	180	172
query54	1233	411	404	404
query55	73	74	77	74
query56	255	223	235	223
query57	1262	1160	1140	1140
query58	228	207	206	206
query59	3175	3072	3043	3043
query60	268	249	246	246
query61	109	108	112	108
query62	859	762	663	663
query63	204	185	183	183
query64	5110	660	655	655
query65	3261	3198	3213	3198
query66	1259	301	308	301
query67	16141	15762	15540	15540
query68	4960	591	582	582
query69	433	248	249	248
query70	1152	1147	1096	1096
query71	388	256	251	251
query72	6186	4100	3999	3999
query73	764	353	359	353
query74	10506	9092	9039	9039
query75	3423	2647	2642	2642
query76	2736	988	1048	988
query77	407	282	279	279
query78	10412	9425	9391	9391
query79	1356	593	578	578
query80	867	422	440	422
query81	554	238	239	238
query82	1319	120	116	116
query83	224	159	163	159
query84	246	71	77	71
query85	1258	311	298	298
query86	341	300	300	300
query87	4942	4706	4687	4687
query88	3338	2195	2154	2154
query89	402	294	284	284
query90	2055	188	191	188
query91	139	107	107	107
query92	60	47	49	47
query93	1440	532	540	532
query94	836	286	295	286
query95	348	243	249	243
query96	615	276	281	276
query97	2925	2697	2704	2697
query98	214	198	197	197
query99	1526	1286	1307	1286
Total cold run time: 306263 ms
Total hot run time: 194481 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b735f7cc1fe1d9cc71a26b210986b6f5ae336377, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.04	0.03
query3	0.23	0.06	0.07
query4	1.64	0.10	0.11
query5	0.42	0.41	0.40
query6	1.19	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.56	0.50	0.50
query10	0.53	0.53	0.55
query11	0.13	0.10	0.11
query12	0.14	0.13	0.11
query13	0.62	0.60	0.61
query14	2.70	2.82	2.73
query15	0.90	0.83	0.83
query16	0.37	0.38	0.39
query17	1.08	1.02	1.04
query18	0.20	0.20	0.20
query19	1.92	1.79	1.92
query20	0.01	0.01	0.01
query21	15.36	0.63	0.60
query22	2.52	2.14	1.63
query23	16.89	0.99	0.79
query24	3.88	0.60	1.49
query25	0.17	0.26	0.04
query26	0.53	0.14	0.13
query27	0.05	0.07	0.04
query28	10.18	1.11	1.07
query29	12.58	3.38	3.38
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.27	0.46	0.45
query33	3.03	3.07	3.02
query34	17.07	4.49	4.43
query35	4.47	4.47	4.50
query36	0.65	0.47	0.46
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.03 s
Total hot run time: 31.97 s

Co-authored-by: Xin Liao <[email protected]>
@kaijchen
Copy link
Contributor Author

run buildall

Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 14, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants