Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix]fix query statistics leak in BE #49325

Merged
merged 1 commit into from
Mar 24, 2025
Merged

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Mar 20, 2025

What problem does this PR solve?

1 clear external query's query statistics when query is finish.
2 add thrift timeout when be report to FE, or report thread may hang.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Mar 20, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wangbo
Copy link
Contributor Author

wangbo commented Mar 20, 2025

run buildall

@yiguolei
Copy link
Contributor

run buildall

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 22, 2025
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei added usercase Important user case type label dev/2.1.x dev/3.0.x labels Mar 22, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 34330 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7be44c591764c4f23ae257309809a31dab3cd8d6, data reload: false

------ Round 1 ----------------------------------
q1	25174	5056	5151	5056
q2	2055	297	156	156
q3	10402	1257	674	674
q4	10238	994	528	528
q5	7520	2374	2362	2362
q6	182	163	136	136
q7	924	733	609	609
q8	9343	1346	1139	1139
q9	6843	5141	5137	5137
q10	6874	2312	1887	1887
q11	498	273	251	251
q12	362	352	212	212
q13	18014	3675	3100	3100
q14	243	242	201	201
q15	531	491	485	485
q16	603	617	582	582
q17	578	851	343	343
q18	7512	7325	7244	7244
q19	1416	961	569	569
q20	321	323	198	198
q21	4005	2701	2473	2473
q22	1073	1045	988	988
Total cold run time: 114711 ms
Total hot run time: 34330 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5134	5116	5076	5076
q2	238	328	234	234
q3	2148	2646	2297	2297
q4	1403	1787	1375	1375
q5	4435	4385	4418	4385
q6	219	167	130	130
q7	2013	1937	1767	1767
q8	2613	2610	2558	2558
q9	7258	7209	6983	6983
q10	3042	3206	2756	2756
q11	580	508	508	508
q12	705	779	615	615
q13	3487	3919	3272	3272
q14	273	325	274	274
q15	520	475	486	475
q16	675	693	627	627
q17	1147	1657	1301	1301
q18	7758	7571	7441	7441
q19	805	831	899	831
q20	1974	2046	1816	1816
q21	5226	4983	4823	4823
q22	1106	1044	1008	1008
Total cold run time: 52759 ms
Total hot run time: 50552 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192820 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7be44c591764c4f23ae257309809a31dab3cd8d6, data reload: false

query1	1465	1075	1039	1039
query2	6242	1921	1870	1870
query3	11144	4672	4526	4526
query4	25653	23822	23014	23014
query5	4407	651	471	471
query6	323	203	193	193
query7	3976	480	267	267
query8	301	243	235	235
query9	8515	2566	2554	2554
query10	470	308	253	253
query11	15320	15029	16248	15029
query12	167	118	112	112
query13	1559	516	401	401
query14	9046	6881	6076	6076
query15	200	195	163	163
query16	7502	625	482	482
query17	1171	722	562	562
query18	2066	403	303	303
query19	183	185	180	180
query20	120	125	121	121
query21	198	121	106	106
query22	4517	4480	4519	4480
query23	33970	33503	33349	33349
query24	7771	2413	2404	2404
query25	495	473	413	413
query26	785	280	149	149
query27	2191	483	322	322
query28	4418	2474	2424	2424
query29	653	584	465	465
query30	282	225	204	204
query31	923	870	812	812
query32	73	64	67	64
query33	549	407	296	296
query34	788	845	514	514
query35	785	853	768	768
query36	963	1010	891	891
query37	124	105	76	76
query38	4173	4157	4227	4157
query39	1493	1424	1446	1424
query40	216	118	106	106
query41	54	53	50	50
query42	156	109	111	109
query43	499	493	489	489
query44	1334	811	798	798
query45	175	170	170	170
query46	839	1040	663	663
query47	1837	1887	1767	1767
query48	369	423	321	321
query49	716	520	423	423
query50	712	744	415	415
query51	4263	4313	4265	4265
query52	116	111	99	99
query53	225	260	182	182
query54	489	500	418	418
query55	88	85	80	80
query56	275	286	265	265
query57	1160	1158	1102	1102
query58	260	248	229	229
query59	2750	2783	2719	2719
query60	317	305	276	276
query61	157	171	122	122
query62	799	748	663	663
query63	217	181	184	181
query64	3195	1034	673	673
query65	4501	4484	4471	4471
query66	859	432	291	291
query67	16401	15479	15327	15327
query68	8599	879	507	507
query69	489	298	257	257
query70	1243	1089	1103	1089
query71	474	277	254	254
query72	5156	4918	4850	4850
query73	679	545	343	343
query74	8949	9081	8937	8937
query75	3951	3217	2715	2715
query76	3666	1177	740	740
query77	785	363	278	278
query78	9847	10028	9268	9268
query79	2740	798	556	556
query80	579	497	450	450
query81	461	252	220	220
query82	190	128	93	93
query83	169	174	156	156
query84	287	95	78	78
query85	818	415	371	371
query86	329	297	295	295
query87	4454	4614	4350	4350
query88	2842	2233	2228	2228
query89	424	311	276	276
query90	2196	215	207	207
query91	147	145	108	108
query92	76	63	55	55
query93	1998	1046	575	575
query94	671	403	376	376
query95	351	262	263	262
query96	490	560	271	271
query97	3297	3344	3303	3303
query98	221	208	204	204
query99	1381	1404	1305	1305
Total cold run time: 275919 ms
Total hot run time: 192820 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7be44c591764c4f23ae257309809a31dab3cd8d6, data reload: false

query1	0.04	0.04	0.02
query2	0.12	0.10	0.10
query3	0.25	0.19	0.19
query4	1.59	0.18	0.18
query5	0.59	0.58	0.57
query6	1.17	0.74	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.54	0.52
query10	0.58	0.59	0.58
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.60	0.59
query14	2.81	2.84	2.82
query15	0.94	0.85	0.83
query16	0.37	0.37	0.38
query17	1.02	1.05	1.01
query18	0.20	0.20	0.19
query19	1.94	2.01	1.81
query20	0.01	0.01	0.02
query21	15.35	0.87	0.55
query22	0.77	0.94	0.73
query23	15.05	1.38	0.60
query24	7.08	1.74	1.38
query25	0.49	0.31	0.06
query26	0.60	0.15	0.13
query27	0.05	0.05	0.05
query28	9.69	0.85	0.41
query29	12.57	4.02	3.37
query30	0.27	0.09	0.06
query31	2.81	0.58	0.39
query32	3.22	0.54	0.47
query33	2.96	3.01	3.06
query34	15.74	5.19	4.55
query35	4.55	4.56	4.53
query36	0.67	0.50	0.49
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.5 s
Total hot run time: 31.88 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/24) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 49.97% (13372/26760)
Line Coverage 39.43% (115698/293406)
Region Coverage 38.13% (58717/154008)
Branch Coverage 33.26% (29633/89088)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.9-merged dev/3.0.5-merged reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants