Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement] (nereids)implement showBackendsCommand in nereids #43121

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

[Enhancement] (nereids)implement showBackendsCommand in nereids #43121

wants to merge 2 commits into from
+129 −1

Conversation

Vallishp
Copy link
Contributor

@Vallishp Vallishp commented Nov 1, 2024

What problem does this PR solve?

Support Show Backends in nereids

Issue Number: close #42762

Related PR: #xxx

Problem Summary:

Check List (For Committer)

  • Test

    • Regression test yes
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

Already testcase present test_backend.groovy
RemoteUsedCapacity:0.000 , BackendId:10002, CpuCores:8, LastStartTime:2024-11-01 17:45:25, Tag:{"location" : "default"}]]
2024-11-01 17:49:17.647 INFO [non-concurrent-thread-1] (Suite.groovy:384) - Execute sql: ALTER SYSTEM DECOMMISSION BACKEND "10002"
2024-11-01 17:49:17.666 INFO [non-concurrent-thread-1] (test_backend.groovy:60) - ALTER SYSTEM DECOMMISSION BACKEND [[0]]
2024-11-01 17:49:17.667 INFO [non-concurrent-thread-1] (Suite.groovy:451) - Execute sql: SHOW BACKENDS;
2024-11-01 17:49:17.694 INFO [non-concurrent-thread-1] (Suite.groovy:384) - Execute sql: CANCEL DECOMMISSION BACKEND "10002"
2024-11-01 17:49:17.707 INFO [non-concurrent-thread-1] (test_backend.groovy:71) - CANCEL DECOMMISSION BACKEND [[0]]
2024-11-01 17:49:17.708 INFO [non-concurrent-thread-1] (Suite.groovy:451) - Execute sql: SHOW BACKENDS;
2024-11-01 17:49:17.718 INFO [non-concurrent-thread-1] (Suite.groovy:451) - Execute sql: show frontends
2024-11-01 17:49:17.805 INFO [non-concurrent-thread-1] (DebugPoint.groovy:162) - disable debug point SystemHandler.decommission_no_check_replica_num for FE 172.20.80.1:8030
2024-11-01 17:49:17.819 INFO [non-concurrent-thread-1] (Http.groovy:85) - http post url=http://172.20.80.1:8030/api/debug_point/remove/SystemHandler.decommission_no_check_replica_num, data=null, isJson=true, response code=200, text={"msg":"success","code":0,"data":null,"count":0}
2024-11-01 17:49:17.820 INFO [non-concurrent-thread-1] (Suite.groovy:384) - Execute sql: admin set frontend config("drop_backend_after_decommission" = "true")
2024-11-01 17:49:17.825 INFO [non-concurrent-thread-1] (ScriptContext.groovy:120) - Run test_backend in /root/doris/workspace/doris/regression-test/suites/node_p0/test_backend.groovy succeed
2024-11-01 17:49:25.546 INFO [main] (RegressionTest.groovy:410) - Success suites:
/root/doris/workspace/doris/regression-test/suites/node_p0/test_backend.groovy: group=nonConcurrent,p0, name=test_backend
2024-11-01 17:49:25.547 INFO [main] (RegressionTest.groovy:492) - All suites success.

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Vallishp
Copy link
Contributor Author

Vallishp commented Nov 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41290 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a673452c8eae97e1a9f48c933991425120bca99b, data reload: false

------ Round 1 ----------------------------------
q1	17627	7496	7318	7318
q2	2049	163	155	155
q3	10637	1068	1178	1068
q4	10537	851	775	775
q5	7762	3060	3069	3060
q6	230	144	146	144
q7	1014	611	610	610
q8	9353	1986	1997	1986
q9	6589	6414	6479	6414
q10	7058	2466	2415	2415
q11	472	264	249	249
q12	421	216	216	216
q13	17775	3044	3002	3002
q14	237	226	205	205
q15	562	536	507	507
q16	664	585	585	585
q17	971	520	545	520
q18	7295	6722	6709	6709
q19	1338	1008	1048	1008
q20	473	181	180	180
q21	4025	3219	3164	3164
q22	1106	1000	1017	1000
Total cold run time: 108195 ms
Total hot run time: 41290 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7376	7287	7316	7287
q2	332	225	227	225
q3	2999	2986	2952	2952
q4	2100	1886	1843	1843
q5	5757	5816	5835	5816
q6	228	141	146	141
q7	2215	1838	1812	1812
q8	3441	3528	3503	3503
q9	8947	8929	8957	8929
q10	3620	3562	3562	3562
q11	593	514	488	488
q12	843	610	621	610
q13	9884	3213	3214	3213
q14	304	293	280	280
q15	571	535	534	534
q16	695	672	656	656
q17	1847	1654	1629	1629
q18	8257	7849	7841	7841
q19	1729	1484	1457	1457
q20	2108	1887	1862	1862
q21	5632	5444	5414	5414
q22	1135	1072	1047	1047
Total cold run time: 70613 ms
Total hot run time: 61101 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196793 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a673452c8eae97e1a9f48c933991425120bca99b, data reload: false

query1	1210	739	728	728
query2	6216	2220	2102	2102
query3	10817	3995	4099	3995
query4	67500	29401	23670	23670
query5	4958	441	423	423
query6	399	159	161	159
query7	5595	290	285	285
query8	316	224	225	224
query9	8834	2735	2736	2735
query10	451	264	263	263
query11	17206	15522	15784	15522
query12	153	105	102	102
query13	1527	445	413	413
query14	10378	6591	6958	6591
query15	202	173	181	173
query16	6822	451	495	451
query17	1102	562	562	562
query18	1719	313	303	303
query19	197	148	144	144
query20	113	106	105	105
query21	196	101	99	99
query22	4599	4463	4411	4411
query23	34596	34361	34294	34294
query24	6004	2855	2835	2835
query25	509	403	397	397
query26	643	158	155	155
query27	1662	279	279	279
query28	4346	2423	2402	2402
query29	687	429	421	421
query30	228	163	149	149
query31	990	804	816	804
query32	69	56	58	56
query33	394	269	266	266
query34	897	534	547	534
query35	842	743	727	727
query36	1068	953	968	953
query37	117	78	70	70
query38	4382	4271	4332	4271
query39	1473	1428	1437	1428
query40	201	101	99	99
query41	49	50	50	50
query42	108	100	100	100
query43	558	505	500	500
query44	1158	827	817	817
query45	182	169	170	169
query46	1132	731	684	684
query47	2001	1831	1876	1831
query48	416	316	318	316
query49	742	387	408	387
query50	807	399	404	399
query51	7282	7332	7273	7273
query52	101	87	88	87
query53	254	182	180	180
query54	521	408	404	404
query55	81	77	75	75
query56	257	258	257	257
query57	1300	1233	1160	1160
query58	207	193	223	193
query59	3447	3274	3310	3274
query60	272	241	250	241
query61	108	108	106	106
query62	778	667	684	667
query63	220	189	186	186
query64	1338	644	608	608
query65	3245	3183	3240	3183
query66	708	305	325	305
query67	16062	15640	15611	15611
query68	3519	578	556	556
query69	407	248	288	248
query70	1143	1088	1174	1088
query71	352	251	241	241
query72	6358	3986	3961	3961
query73	766	346	350	346
query74	9981	9080	9121	9080
query75	3369	2645	2662	2645
query76	1868	1098	1034	1034
query77	473	272	271	271
query78	10318	9417	9441	9417
query79	1504	589	596	589
query80	877	419	410	410
query81	520	238	237	237
query82	1295	114	120	114
query83	225	137	137	137
query84	281	69	71	69
query85	896	305	286	286
query86	354	352	291	291
query87	4897	4821	4793	4793
query88	3716	2179	2139	2139
query89	416	287	287	287
query90	2005	185	187	185
query91	139	99	108	99
query92	61	49	52	49
query93	1929	545	541	541
query94	785	313	291	291
query95	360	251	246	246
query96	612	278	283	278
query97	2931	2722	2695	2695
query98	208	199	198	198
query99	1605	1329	1309	1309
Total cold run time: 317122 ms
Total hot run time: 196793 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a673452c8eae97e1a9f48c933991425120bca99b, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.64	0.10	0.10
query5	0.42	0.41	0.41
query6	1.17	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.49	0.48
query10	0.54	0.55	0.56
query11	0.14	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	2.81	2.73	2.73
query15	0.89	0.82	0.83
query16	0.37	0.38	0.40
query17	1.02	0.98	1.02
query18	0.20	0.19	0.19
query19	1.82	1.89	1.91
query20	0.01	0.02	0.01
query21	15.36	0.60	0.58
query22	2.69	1.63	2.08
query23	17.08	0.95	0.82
query24	3.83	0.44	1.66
query25	0.22	0.16	0.18
query26	0.42	0.13	0.13
query27	0.04	0.05	0.04
query28	10.32	1.10	1.07
query29	12.54	3.25	3.22
query30	0.24	0.05	0.07
query31	2.87	0.39	0.39
query32	3.28	0.46	0.45
query33	2.94	3.03	3.00
query34	16.82	4.52	4.48
query35	4.50	4.52	4.46
query36	0.69	0.48	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.02
query40	0.15	0.12	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.01 s
Total hot run time: 31.85 s

@Vallishp Vallishp force-pushed the nereids_backends branch 3 times, most recently from d3e1f1c to ffea06f Compare November 14, 2024 05:52
@Vallishp
Copy link
Contributor Author

run buildall

@Vallishp
Copy link
Contributor Author

run buildall

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement showBackendsCommand in nereids
3 participants
@Vallishp @doris-robot @starocean999