Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Outfile) forbid parallel outfile if pipeline engine enabled. #43439

Merged

Conversation

BePPPower
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #43437

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the area/planner Issues or PRs related to the query planner label Nov 7, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 48849 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9dae5490efb026a6da9eb7ddfdda6211a9b8e27, data reload: false

------ Round 1 ----------------------------------
q1	17697	4368	4371	4368
q2	2095	149	148	148
q3	10509	1953	1911	1911
q4	10321	1265	1317	1265
q5	8838	3891	3924	3891
q6	238	121	125	121
q7	2070	1621	1606	1606
q8	9257	2705	2699	2699
q9	10163	9995	9569	9569
q10	8662	3503	3515	3503
q11	432	248	248	248
q12	474	294	303	294
q13	18360	3971	4045	3971
q14	366	317	331	317
q15	507	459	465	459
q16	561	448	459	448
q17	1116	927	926	926
q18	7266	6870	6912	6870
q19	1692	1534	1476	1476
q20	499	307	296	296
q21	4445	4093	4061	4061
q22	491	402	408	402
Total cold run time: 116059 ms
Total hot run time: 48849 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4308	4300	4300
q2	320	218	225	218
q3	4179	4170	4167	4167
q4	2752	2750	2729	2729
q5	7186	7125	7117	7117
q6	236	121	119	119
q7	3218	2813	2833	2813
q8	4358	4477	4491	4477
q9	13775	13675	13616	13616
q10	4266	4231	4189	4189
q11	765	682	693	682
q12	1040	860	848	848
q13	6750	3727	3691	3691
q14	464	429	424	424
q15	501	464	455	455
q16	652	596	593	593
q17	3819	3832	3922	3832
q18	8861	8703	8766	8703
q19	1710	1663	1677	1663
q20	2380	2112	2121	2112
q21	8467	8483	8430	8430
q22	1110	962	921	921
Total cold run time: 81115 ms
Total hot run time: 76099 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 211714 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9dae5490efb026a6da9eb7ddfdda6211a9b8e27, data reload: false

query1	945	427	381	381
query2	6575	2216	2242	2216
query3	6925	200	201	200
query4	23333	21250	21973	21250
query5	19748	6485	6520	6485
query6	286	220	235	220
query7	4288	300	302	300
query8	261	275	244	244
query9	3063	2657	2582	2582
query10	450	316	290	290
query11	15995	15325	14893	14893
query12	121	80	72	72
query13	1037	448	438	438
query14	17556	13247	13360	13247
query15	375	224	231	224
query16	6487	288	266	266
query17	1748	918	896	896
query18	894	320	306	306
query19	217	155	160	155
query20	107	100	99	99
query21	189	96	93	93
query22	5299	5128	5091	5091
query23	34223	33350	33391	33350
query24	7796	6289	6290	6289
query25	520	416	432	416
query26	1263	165	158	158
query27	2436	296	290	290
query28	6038	2253	2238	2238
query29	2859	2731	2712	2712
query30	244	169	167	167
query31	923	745	768	745
query32	69	64	59	59
query33	464	273	262	262
query34	867	480	485	480
query35	1116	859	926	859
query36	1253	1510	1151	1151
query37	171	61	61	61
query38	3094	2975	2926	2926
query39	1369	1353	1328	1328
query40	314	98	96	96
query41	41	37	38	37
query42	83	84	77	77
query43	651	648	643	643
query44	1151	723	722	722
query45	243	235	229	229
query46	1232	956	932	932
query47	1914	1887	1845	1845
query48	500	407	423	407
query49	664	387	371	371
query50	868	596	632	596
query51	4761	4618	4591	4591
query52	87	76	83	76
query53	223	185	182	182
query54	2658	2452	2488	2452
query55	82	87	82	82
query56	245	217	209	209
query57	1275	1138	1177	1138
query58	224	197	216	197
query59	3479	3311	3488	3311
query60	220	224	204	204
query61	101	98	100	98
query62	788	493	455	455
query63	198	172	174	172
query64	3534	1590	1516	1516
query65	3624	3549	3539	3539
query66	771	414	389	389
query67	16185	15779	15353	15353
query68	9191	644	638	638
query69	511	270	263	263
query70	1648	1460	1285	1285
query71	418	300	319	300
query72	6747	4887	4690	4690
query73	749	322	309	309
query74	6309	5804	5823	5804
query75	4662	3778	3715	3715
query76	4560	1138	1177	1138
query77	577	257	263	257
query78	12617	11756	11451	11451
query79	6766	627	625	625
query80	2936	395	374	374
query81	532	238	236	236
query82	1481	97	103	97
query83	185	135	133	133
query84	256	70	70	70
query85	1387	324	327	324
query86	347	294	288	288
query87	3184	3038	2989	2989
query88	4687	2287	2300	2287
query89	399	294	291	291
query90	1748	218	216	216
query91	160	127	124	124
query92	58	51	50	50
query93	4959	553	565	553
query94	858	209	207	207
query95	2103	1967	1944	1944
query96	631	323	329	323
query97	6373	6326	6266	6266
query98	236	199	205	199
query99	2864	910	939	910
Total cold run time: 316859 ms
Total hot run time: 211714 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9dae5490efb026a6da9eb7ddfdda6211a9b8e27, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.76	0.06	0.08
query5	0.54	0.52	0.52
query6	1.22	0.72	0.61
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.53	0.49	0.47
query10	0.54	0.53	0.53
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.61	0.61
query14	0.79	0.79	0.79
query15	0.78	0.75	0.75
query16	0.36	0.36	0.35
query17	1.05	1.01	1.03
query18	0.23	0.25	0.24
query19	1.94	1.87	1.85
query20	0.02	0.01	0.01
query21	15.47	0.56	0.53
query22	2.03	1.85	1.73
query23	17.26	1.06	1.10
query24	4.89	0.89	1.32
query25	0.39	0.11	0.05
query26	0.53	0.15	0.16
query27	0.04	0.05	0.03
query28	8.04	0.74	0.82
query29	12.67	2.31	2.39
query30	0.56	0.53	0.53
query31	2.80	0.39	0.36
query32	3.40	0.51	0.49
query33	3.08	3.09	3.04
query34	15.26	4.82	4.79
query35	4.85	4.88	4.83
query36	1.09	1.02	1.01
query37	0.05	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.15
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 103.74 s
Total hot run time: 30.89 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e9dae5490efb026a6da9eb7ddfdda6211a9b8e27 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.7 seconds inserted 10000000 Rows, about 460K ops/s

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 11, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 81f43a1 into apache:branch-2.0 Nov 11, 2024
23 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. area/planner Issues or PRs related to the query planner dev/2.0.16-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants