Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feat](nereids) support ShowTabletStorageFormat command in nereids #43295

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

rijeshkp
Copy link
Contributor

@rijeshkp rijeshkp commented Nov 5, 2024

What problem does this PR solve?

Issue Number: close #42783

Related PR: #xxx

Problem Summary:

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.
  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from a1fb1da to f3729c0 Compare November 6, 2024 02:50
@rijeshkp rijeshkp marked this pull request as ready for review November 6, 2024 02:55
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41191 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c36ee4bab9227ccc21763754c22a01c4542ac89f, data reload: false

------ Round 1 ----------------------------------
q1	17619	7893	7337	7337
q2	2030	166	160	160
q3	10686	1071	1145	1071
q4	10548	884	820	820
q5	7723	3095	3026	3026
q6	236	145	145	145
q7	1003	607	596	596
q8	9350	1987	1963	1963
q9	6601	6460	6484	6460
q10	7127	2399	2429	2399
q11	474	263	261	261
q12	416	215	210	210
q13	17799	3048	3029	3029
q14	246	223	213	213
q15	572	528	524	524
q16	657	572	580	572
q17	973	532	497	497
q18	7247	6755	6826	6755
q19	1334	1096	921	921
q20	446	184	181	181
q21	3972	3172	3061	3061
q22	1120	999	990	990
Total cold run time: 108179 ms
Total hot run time: 41191 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7292	7253	7219	7219
q2	318	230	228	228
q3	3020	2949	2974	2949
q4	2076	1957	1816	1816
q5	5711	5761	5799	5761
q6	233	138	143	138
q7	2229	1833	1824	1824
q8	3429	3501	3453	3453
q9	8982	8945	8877	8877
q10	3616	3551	3565	3551
q11	600	518	508	508
q12	820	642	610	610
q13	9972	3211	3191	3191
q14	324	290	297	290
q15	577	530	515	515
q16	677	634	636	634
q17	1856	1618	1598	1598
q18	8250	7881	7697	7697
q19	1715	1655	1610	1610
q20	2143	1863	1893	1863
q21	5651	5485	5553	5485
q22	1187	1103	1068	1068
Total cold run time: 70678 ms
Total hot run time: 60885 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197783 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c36ee4bab9227ccc21763754c22a01c4542ac89f, data reload: false

query1	1183	916	958	916
query2	6233	2080	2028	2028
query3	10892	4161	4157	4157
query4	67427	28534	24122	24122
query5	5053	468	423	423
query6	389	161	169	161
query7	5640	282	282	282
query8	296	207	213	207
query9	9015	2611	2604	2604
query10	473	261	262	261
query11	17523	15373	16006	15373
query12	154	100	104	100
query13	1579	428	442	428
query14	10447	7545	7323	7323
query15	219	198	179	179
query16	7124	527	510	510
query17	1046	562	569	562
query18	1845	301	308	301
query19	195	155	153	153
query20	120	111	108	108
query21	204	102	103	102
query22	4740	4466	4422	4422
query23	34907	34492	34354	34354
query24	6052	2827	2714	2714
query25	519	402	396	396
query26	658	160	158	158
query27	1706	295	284	284
query28	3873	2430	2407	2407
query29	694	430	433	430
query30	234	160	154	154
query31	958	817	802	802
query32	66	53	57	53
query33	411	267	274	267
query34	909	520	504	504
query35	849	744	729	729
query36	1064	959	973	959
query37	116	76	73	73
query38	4313	4365	4282	4282
query39	1446	1429	1417	1417
query40	205	100	101	100
query41	46	45	45	45
query42	108	94	97	94
query43	522	486	486	486
query44	1163	812	817	812
query45	182	167	168	167
query46	1139	707	693	693
query47	1981	1882	1880	1880
query48	422	322	318	318
query49	750	426	391	391
query50	804	393	398	393
query51	7218	7213	7176	7176
query52	99	89	84	84
query53	253	177	183	177
query54	520	442	382	382
query55	83	71	77	71
query56	246	237	235	235
query57	1281	1176	1143	1143
query58	224	219	205	205
query59	3166	3050	3104	3050
query60	271	244	241	241
query61	104	101	100	100
query62	793	685	671	671
query63	208	184	185	184
query64	1364	614	594	594
query65	3353	3224	3270	3224
query66	738	322	299	299
query67	16136	15862	15619	15619
query68	3233	567	589	567
query69	411	255	255	255
query70	1174	1158	1148	1148
query71	359	247	251	247
query72	6100	4069	4049	4049
query73	760	355	352	352
query74	10020	9029	9024	9024
query75	3425	2679	2675	2675
query76	1570	992	1123	992
query77	495	283	282	282
query78	10460	9503	9809	9503
query79	1349	593	595	593
query80	813	424	424	424
query81	512	237	241	237
query82	1316	116	121	116
query83	234	135	143	135
query84	282	74	65	65
query85	875	296	293	293
query86	340	298	307	298
query87	4912	4731	4695	4695
query88	3355	2210	2157	2157
query89	416	293	287	287
query90	1969	187	181	181
query91	133	99	100	99
query92	60	47	49	47
query93	1375	535	530	530
query94	775	295	294	294
query95	343	243	251	243
query96	613	271	278	271
query97	2922	2723	2695	2695
query98	211	197	207	197
query99	1787	1304	1293	1293
Total cold run time: 316550 ms
Total hot run time: 197783 ms

if (result == null) {
throw new AnalysisException("get tablet data from backend: " + be.getId() + "error.");
}
if (isVerbose()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove isVerbose() function, access verbose member directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed

@Override
public void run(ConnectContext ctx, StmtExecutor executor) throws Exception {
if (!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.ADMIN)
&& !Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

!Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ConnectContext.get(), PrivPredicate.OPERATOR) is not in ShowTabletStorageFormatStmt's analyze() method, why need it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the check. Thanks for pointing it out. Since this is admin only, the operator should not be allowed.

/**
* admin show tablet storage format command
*/
public class ShowTabletStorageFormatCommand extends Command implements NoForward {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a new method checkSupportedInCloudMode() is added in Command.java by https://github.com/apache/doris/pull/43271/files, ShowTabletStorageFormatCommand need implement it with same logic like ShowTabletStorageFormatStmt, please update after the pr is merged

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure. Update after the PR is merged.

@starocean999 starocean999 changed the title Fixes [Feat](nereids) support ShowTabletStorageFormat command in nereids Nov 6, 2024
*/
public class ShowTabletStorageFormatCommand extends Command implements NoForward {
public static final Logger LOG = LogManager.getLogger(ShowTabletStorageFormatCommand.class);
private boolean verbose;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private boolean verbose;
private final boolean verbose;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@rijeshkp rijeshkp force-pushed the ShowTableStorageFormatToNereids branch from c36ee4b to 53323be Compare November 6, 2024 14:31
@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

run buildall

@rijeshkp
Copy link
Contributor Author

rijeshkp commented Nov 6, 2024

Output Screenshot

image

@doris-robot
Copy link

TPC-H: Total hot run time: 41189 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

------ Round 1 ----------------------------------
q1	17570	8011	7395	7395
q2	2042	171	176	171
q3	10757	1087	1123	1087
q4	10567	810	875	810
q5	7739	3079	3067	3067
q6	231	148	151	148
q7	1005	609	604	604
q8	9371	1977	2008	1977
q9	6685	6453	6438	6438
q10	7093	2441	2412	2412
q11	468	260	259	259
q12	425	215	221	215
q13	17811	2994	3004	2994
q14	257	220	212	212
q15	586	526	518	518
q16	681	591	607	591
q17	969	603	487	487
q18	7300	6595	6647	6595
q19	1324	1036	995	995
q20	462	186	174	174
q21	3930	3197	3061	3061
q22	1122	1007	979	979
Total cold run time: 108395 ms
Total hot run time: 41189 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7290	7228	7305	7228
q2	347	254	244	244
q3	2995	2924	2995	2924
q4	2093	1867	1778	1778
q5	5750	5780	5784	5780
q6	237	147	147	147
q7	2270	1882	1778	1778
q8	3389	3574	3502	3502
q9	8977	8954	8914	8914
q10	3579	3586	3548	3548
q11	605	500	511	500
q12	800	654	648	648
q13	10980	3237	3223	3223
q14	302	274	275	274
q15	616	544	550	544
q16	678	648	638	638
q17	1845	1632	1592	1592
q18	8389	8319	7961	7961
q19	1692	1506	1636	1506
q20	2095	1909	1873	1873
q21	5582	5400	5347	5347
q22	1161	1078	1044	1044
Total cold run time: 71672 ms
Total hot run time: 60993 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196541 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

query1	2256	2057	2076	2057
query2	6215	2046	2041	2041
query3	15513	12013	248	248
query4	33063	23750	23948	23750
query5	3686	456	442	442
query6	271	176	175	175
query7	3983	294	293	293
query8	291	230	233	230
query9	9434	2715	2703	2703
query10	461	249	246	246
query11	18021	15262	15518	15262
query12	155	107	103	103
query13	1579	417	419	417
query14	10265	7395	7266	7266
query15	262	197	196	196
query16	8047	487	518	487
query17	1536	590	618	590
query18	2474	625	663	625
query19	338	186	187	186
query20	128	120	123	120
query21	217	106	102	102
query22	4650	4494	4522	4494
query23	35413	34517	34687	34517
query24	11345	3382	3418	3382
query25	544	400	405	400
query26	733	182	186	182
query27	1923	283	289	283
query28	6598	2436	2441	2436
query29	672	424	428	424
query30	394	311	298	298
query31	1026	783	821	783
query32	99	56	58	56
query33	691	289	286	286
query34	917	516	517	516
query35	882	753	731	731
query36	1106	953	967	953
query37	123	78	73	73
query38	4426	4309	4368	4309
query39	1464	1428	1420	1420
query40	201	105	105	105
query41	48	49	48	48
query42	110	101	99	99
query43	523	501	501	501
query44	1226	813	813	813
query45	185	163	167	163
query46	1154	725	708	708
query47	1923	1827	1841	1827
query48	417	327	320	320
query49	906	409	405	405
query50	823	403	400	400
query51	7225	6999	6989	6989
query52	101	93	88	88
query53	264	186	186	186
query54	1133	434	402	402
query55	83	77	77	77
query56	256	233	241	233
query57	1262	1163	1123	1123
query58	229	206	198	198
query59	3246	3192	3180	3180
query60	273	253	247	247
query61	132	119	112	112
query62	896	664	660	660
query63	212	195	191	191
query64	3791	645	607	607
query65	3350	3241	3259	3241
query66	878	312	321	312
query67	15975	15863	15742	15742
query68	4411	585	581	581
query69	435	248	255	248
query70	1198	1153	1148	1148
query71	350	261	248	248
query72	6029	4054	4047	4047
query73	800	355	353	353
query74	10373	9204	9076	9076
query75	3455	2712	2683	2683
query76	2795	1110	1151	1110
query77	391	277	288	277
query78	11181	9916	9452	9452
query79	2495	621	619	619
query80	936	423	441	423
query81	576	240	241	240
query82	733	118	115	115
query83	245	166	163	163
query84	238	75	70	70
query85	815	306	301	301
query86	429	299	296	296
query87	4948	4695	4711	4695
query88	3902	2152	2142	2142
query89	420	289	294	289
query90	2068	191	188	188
query91	139	115	102	102
query92	57	50	49	49
query93	2260	546	549	546
query94	850	299	303	299
query95	346	254	258	254
query96	635	282	276	276
query97	2879	2704	2741	2704
query98	234	200	191	191
query99	1725	1297	1305	1297
Total cold run time: 306371 ms
Total hot run time: 196541 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 53323bec0ca782af43271b824c6c6778e334d40c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.65	0.10	0.11
query5	0.42	0.40	0.39
query6	1.16	0.65	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.57	0.49	0.51
query10	0.54	0.55	0.56
query11	0.14	0.10	0.11
query12	0.14	0.12	0.10
query13	0.61	0.60	0.59
query14	2.71	2.74	2.85
query15	0.87	0.83	0.83
query16	0.39	0.38	0.36
query17	1.06	0.98	0.99
query18	0.20	0.19	0.20
query19	1.83	1.83	1.85
query20	0.02	0.01	0.01
query21	15.36	0.59	0.59
query22	2.88	1.78	1.66
query23	17.00	0.92	0.83
query24	3.04	1.94	1.38
query25	0.29	0.22	0.12
query26	0.42	0.14	0.13
query27	0.04	0.04	0.05
query28	10.07	1.09	1.06
query29	12.81	3.24	3.26
query30	0.25	0.07	0.06
query31	2.87	0.38	0.37
query32	3.27	0.46	0.46
query33	2.94	3.05	3.05
query34	17.17	4.47	4.51
query35	4.50	4.55	4.53
query36	0.64	0.49	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.73 s
Total hot run time: 32.84 s

@GentleCold
Copy link
Contributor

GentleCold commented Nov 10, 2024

pr's related issue is SHOW TABLET STORAGE FORMAT VERBOSE?, but the commits add the ADMIN SHOW TABLET STORAGE FORMAT VERBOSE? linked here: #42849

@starocean999
Copy link
Contributor

please rebase latest master and resolve the conflicts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids)implement showTabletStorageFormatCommand in nereids
4 participants