Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](udf) support load data with udf functions #43029

Merged
merged 5 commits into from
Nov 6, 2024
Merged

[improve](udf) support load data with udf functions #43029

merged 5 commits into from
Nov 6, 2024

Conversation

zhangstar333
Copy link
Contributor

@zhangstar333 zhangstar333 commented Oct 31, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

support load data with udf functions, eg stream load or routine load use function convert input data

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41406 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0459b7acd9035ad7b27566848184e08483f5737, data reload: false

------ Round 1 ----------------------------------
q1	17831	7714	7338	7338
q2	2054	180	151	151
q3	10636	1140	1179	1140
q4	10282	846	802	802
q5	7737	3462	3091	3091
q6	232	151	145	145
q7	1043	603	600	600
q8	9340	1951	2084	1951
q9	6685	6464	6504	6464
q10	7070	2435	2485	2435
q11	446	243	245	243
q12	409	216	209	209
q13	17787	3003	3036	3003
q14	244	208	213	208
q15	574	532	511	511
q16	640	594	589	589
q17	989	498	537	498
q18	7425	6762	6821	6762
q19	1332	1092	988	988
q20	492	181	176	176
q21	4044	3265	3110	3110
q22	1100	1028	992	992
Total cold run time: 108392 ms
Total hot run time: 41406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7405	7299	7265	7265
q2	323	226	224	224
q3	2947	2734	2798	2734
q4	1953	1688	1752	1688
q5	5492	5521	5607	5521
q6	220	138	136	136
q7	2119	1694	1732	1694
q8	3252	3430	3452	3430
q9	8572	8616	8578	8578
q10	3540	3483	3454	3454
q11	582	473	489	473
q12	785	573	607	573
q13	9815	3035	3003	3003
q14	297	261	280	261
q15	562	516	510	510
q16	697	630	644	630
q17	1865	1609	1575	1575
q18	7873	7426	7581	7426
q19	1681	1615	1655	1615
q20	2086	1828	1811	1811
q21	5448	5287	5304	5287
q22	1104	1001	1006	1001
Total cold run time: 68618 ms
Total hot run time: 58889 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191574 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0459b7acd9035ad7b27566848184e08483f5737, data reload: false

query1	953	376	361	361
query2	5664	2114	2073	2073
query3	6740	211	213	211
query4	34008	23514	23529	23514
query5	4300	449	456	449
query6	258	175	175	175
query7	4585	298	287	287
query8	282	227	227	227
query9	9489	2709	2692	2692
query10	483	256	244	244
query11	18753	15227	15224	15224
query12	153	103	104	103
query13	1692	419	422	419
query14	9611	7053	6921	6921
query15	233	175	182	175
query16	7183	444	431	431
query17	1509	552	532	532
query18	1996	294	287	287
query19	363	150	146	146
query20	113	111	109	109
query21	204	102	101	101
query22	4616	4449	4368	4368
query23	34925	34249	34580	34249
query24	11392	2766	2787	2766
query25	691	396	395	395
query26	1717	157	160	157
query27	2764	284	293	284
query28	8192	2478	2469	2469
query29	889	424	426	424
query30	326	163	157	157
query31	1028	822	834	822
query32	66	57	56	56
query33	714	268	278	268
query34	924	506	504	504
query35	868	742	729	729
query36	1102	947	946	946
query37	140	73	72	72
query38	4305	4265	4224	4224
query39	1484	1406	1393	1393
query40	280	100	102	100
query41	49	47	48	47
query42	113	101	98	98
query43	521	500	472	472
query44	1265	807	803	803
query45	176	166	163	163
query46	1135	684	676	676
query47	1963	1875	1844	1844
query48	433	342	311	311
query49	1153	392	386	386
query50	804	388	388	388
query51	7332	7032	7222	7032
query52	99	89	89	89
query53	260	186	176	176
query54	1059	407	408	407
query55	78	76	77	76
query56	254	243	239	239
query57	1303	1177	1141	1141
query58	234	209	210	209
query59	3219	2978	2936	2936
query60	281	252	259	252
query61	120	117	125	117
query62	861	676	669	669
query63	218	190	189	189
query64	3785	705	596	596
query65	3292	3225	3218	3218
query66	1130	309	312	309
query67	16438	15759	15763	15759
query68	4646	560	575	560
query69	398	254	245	245
query70	1200	1129	1043	1043
query71	352	244	250	244
query72	6153	4040	4039	4039
query73	794	358	357	357
query74	10052	8943	9009	8943
query75	3431	2678	2674	2674
query76	2098	1024	1071	1024
query77	405	277	267	267
query78	10613	9587	9473	9473
query79	1131	598	590	590
query80	706	418	419	418
query81	521	236	232	232
query82	684	118	118	118
query83	248	134	136	134
query84	230	72	65	65
query85	940	316	286	286
query86	315	300	271	271
query87	4918	4679	4587	4587
query88	3460	2225	2170	2170
query89	406	297	295	295
query90	1959	183	179	179
query91	133	100	101	100
query92	56	46	48	46
query93	1066	542	550	542
query94	718	290	280	280
query95	354	250	240	240
query96	612	287	277	277
query97	2883	2760	2716	2716
query98	206	201	186	186
query99	1508	1307	1324	1307
Total cold run time: 295766 ms
Total hot run time: 191574 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0459b7acd9035ad7b27566848184e08483f5737, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.07	0.07
query4	1.63	0.10	0.10
query5	0.42	0.39	0.40
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.49	0.50
query10	0.55	0.55	0.55
query11	0.14	0.10	0.12
query12	0.14	0.10	0.11
query13	0.61	0.60	0.59
query14	2.75	2.83	2.71
query15	0.89	0.82	0.83
query16	0.38	0.37	0.37
query17	1.06	1.06	1.04
query18	0.20	0.20	0.20
query19	1.86	1.83	1.95
query20	0.01	0.01	0.01
query21	15.36	0.58	0.58
query22	2.64	2.20	2.39
query23	17.66	0.87	0.96
query24	3.16	1.80	1.40
query25	0.15	0.20	0.08
query26	0.49	0.14	0.13
query27	0.05	0.04	0.04
query28	9.78	1.11	1.07
query29	12.54	3.28	3.24
query30	0.24	0.06	0.06
query31	2.88	0.38	0.39
query32	3.28	0.45	0.45
query33	2.95	3.03	3.01
query34	16.92	4.44	4.40
query35	4.45	4.47	4.50
query36	0.64	0.48	0.49
query37	0.08	0.05	0.06
query38	0.05	0.03	0.03
query39	0.03	0.03	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 106.41 s
Total hot run time: 33.25 s

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41108 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 530bca17158f783abba81f05194d53bd485acf35, data reload: false

------ Round 1 ----------------------------------
q1	17559	7425	7334	7334
q2	2052	163	157	157
q3	10635	1079	1213	1079
q4	10567	833	888	833
q5	7740	3043	3049	3043
q6	234	144	143	143
q7	1007	607	611	607
q8	9335	1932	1986	1932
q9	6625	6406	6374	6374
q10	7032	2419	2414	2414
q11	458	260	259	259
q12	406	226	217	217
q13	17769	3008	3015	3008
q14	236	227	210	210
q15	555	521	510	510
q16	627	584	574	574
q17	968	532	547	532
q18	7172	6563	6655	6563
q19	1340	1080	1035	1035
q20	463	171	175	171
q21	3960	3175	3109	3109
q22	1106	1035	1004	1004
Total cold run time: 107846 ms
Total hot run time: 41108 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7302	7257	7282	7257
q2	319	222	222	222
q3	2967	2943	2973	2943
q4	2036	1911	1821	1821
q5	5758	5698	5769	5698
q6	224	144	143	143
q7	2258	1848	1842	1842
q8	3357	3593	3489	3489
q9	8922	8925	8898	8898
q10	3576	3560	3532	3532
q11	620	508	508	508
q12	864	634	567	567
q13	11008	3187	3207	3187
q14	305	287	276	276
q15	597	528	516	516
q16	675	639	654	639
q17	1839	1588	1566	1566
q18	8296	7695	7654	7654
q19	1716	1550	1627	1550
q20	2113	1886	1894	1886
q21	5784	5479	5374	5374
q22	1165	1067	1048	1048
Total cold run time: 71701 ms
Total hot run time: 60616 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196080 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 530bca17158f783abba81f05194d53bd485acf35, data reload: false

query1	1216	917	925	917
query2	6217	2059	2042	2042
query3	10818	3807	4013	3807
query4	67853	29480	23579	23579
query5	4949	469	449	449
query6	410	173	164	164
query7	5649	291	300	291
query8	321	226	229	226
query9	8990	2755	2736	2736
query10	466	256	254	254
query11	17546	15416	15862	15416
query12	153	105	102	102
query13	1564	463	434	434
query14	10804	7322	7278	7278
query15	202	186	196	186
query16	7032	481	487	481
query17	1023	552	549	549
query18	1781	295	292	292
query19	230	151	150	150
query20	117	112	115	112
query21	202	102	104	102
query22	4792	4475	4273	4273
query23	34619	33919	34155	33919
query24	6000	2740	2665	2665
query25	505	395	416	395
query26	645	158	157	157
query27	1696	287	291	287
query28	4239	2467	2437	2437
query29	684	426	428	426
query30	239	160	151	151
query31	1003	801	821	801
query32	67	62	59	59
query33	418	267	277	267
query34	901	509	511	509
query35	881	748	744	744
query36	1074	949	962	949
query37	116	78	73	73
query38	4375	4291	4265	4265
query39	1471	1430	1409	1409
query40	202	100	103	100
query41	47	45	46	45
query42	107	100	100	100
query43	537	505	502	502
query44	1160	798	816	798
query45	187	167	164	164
query46	1108	679	694	679
query47	1944	1846	1876	1846
query48	427	315	328	315
query49	747	401	396	396
query50	814	400	398	398
query51	7280	7111	7044	7044
query52	103	89	90	89
query53	255	190	190	190
query54	524	398	405	398
query55	76	75	77	75
query56	292	243	238	238
query57	1307	1179	1133	1133
query58	209	205	203	203
query59	3213	2992	2941	2941
query60	272	250	248	248
query61	102	100	105	100
query62	782	653	673	653
query63	217	190	184	184
query64	1342	626	624	624
query65	3275	3227	3239	3227
query66	700	296	298	296
query67	15991	15752	15651	15651
query68	3555	605	590	590
query69	421	255	250	250
query70	1163	1164	1099	1099
query71	353	259	286	259
query72	6217	3923	3947	3923
query73	749	365	361	361
query74	10111	8984	9174	8984
query75	3399	2676	2685	2676
query76	1773	1009	1108	1009
query77	486	284	271	271
query78	10491	9468	9461	9461
query79	2088	607	604	604
query80	1385	422	428	422
query81	537	242	242	242
query82	1289	124	119	119
query83	258	138	149	138
query84	291	75	66	66
query85	1029	294	285	285
query86	410	306	301	301
query87	4878	4692	4743	4692
query88	3645	2217	2142	2142
query89	417	288	290	288
query90	1946	189	187	187
query91	132	99	101	99
query92	67	50	48	48
query93	2618	558	553	553
query94	871	291	295	291
query95	347	246	249	246
query96	630	280	286	280
query97	2924	2729	2712	2712
query98	210	205	195	195
query99	1594	1306	1297	1297
Total cold run time: 320772 ms
Total hot run time: 196080 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 530bca17158f783abba81f05194d53bd485acf35, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.40	0.39	0.41
query6	1.17	0.66	0.66
query7	0.02	0.01	0.03
query8	0.04	0.02	0.03
query9	0.57	0.49	0.50
query10	0.53	0.55	0.57
query11	0.14	0.10	0.11
query12	0.12	0.11	0.11
query13	0.61	0.58	0.59
query14	2.73	2.73	2.79
query15	0.92	0.82	0.81
query16	0.41	0.39	0.39
query17	1.06	1.05	1.01
query18	0.21	0.20	0.20
query19	1.88	1.72	1.92
query20	0.01	0.01	0.02
query21	15.36	0.59	0.58
query22	2.61	2.93	1.61
query23	17.04	0.97	0.83
query24	2.98	1.46	1.66
query25	0.24	0.29	0.05
query26	0.40	0.14	0.13
query27	0.05	0.04	0.04
query28	9.87	1.09	1.07
query29	12.54	3.19	3.21
query30	0.24	0.06	0.06
query31	2.88	0.36	0.38
query32	3.30	0.46	0.46
query33	2.98	3.01	3.05
query34	16.91	4.47	4.45
query35	4.53	4.50	4.48
query36	0.69	0.49	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 105.87 s
Total hot run time: 32.55 s

HappenLee
HappenLee previously approved these changes Nov 1, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 1, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Nov 1, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhangstar333 zhangstar333 dismissed stale reviews from dataroaring and HappenLee via 6870801 November 1, 2024 09:28
@zhangstar333
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41599 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6870801f4a0356ee06983f6a36c7477a28975d12, data reload: false

------ Round 1 ----------------------------------
q1	17563	8110	7321	7321
q2	2043	167	188	167
q3	10589	1136	1188	1136
q4	10238	897	778	778
q5	7730	3087	3104	3087
q6	238	148	151	148
q7	1042	621	617	617
q8	9360	1964	2000	1964
q9	6639	6466	6465	6465
q10	7076	2421	2422	2421
q11	458	261	259	259
q12	406	220	212	212
q13	17767	3021	3052	3021
q14	242	215	215	215
q15	580	533	519	519
q16	671	594	610	594
q17	968	523	528	523
q18	7324	6679	6735	6679
q19	1338	1041	1020	1020
q20	480	187	183	183
q21	4000	3267	3331	3267
q22	1116	1014	1003	1003
Total cold run time: 107868 ms
Total hot run time: 41599 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7305	7272	7333	7272
q2	323	227	227	227
q3	2977	2818	2792	2792
q4	1974	1741	1727	1727
q5	5454	5518	5494	5494
q6	217	140	141	140
q7	2116	1724	1732	1724
q8	3242	3404	3425	3404
q9	8639	8665	8611	8611
q10	3516	3480	3469	3469
q11	607	496	499	496
q12	791	580	554	554
q13	10000	2984	3053	2984
q14	316	257	263	257
q15	559	509	513	509
q16	663	652	635	635
q17	1832	1574	1574	1574
q18	7902	7405	7320	7320
q19	1669	1542	1452	1452
q20	2046	1842	1846	1842
q21	5496	5252	5119	5119
q22	1118	1019	1022	1019
Total cold run time: 68762 ms
Total hot run time: 58621 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192506 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6870801f4a0356ee06983f6a36c7477a28975d12, data reload: false

query1	960	379	380	379
query2	6507	2099	2093	2093
query3	6778	212	219	212
query4	34185	23692	23690	23690
query5	4324	456	436	436
query6	255	169	161	161
query7	4595	291	287	287
query8	278	252	228	228
query9	9685	2729	2713	2713
query10	474	247	256	247
query11	18257	15257	15250	15250
query12	146	104	100	100
query13	1659	429	421	421
query14	8962	7155	7423	7155
query15	319	177	183	177
query16	8164	456	498	456
query17	1709	570	555	555
query18	2131	296	295	295
query19	380	148	149	148
query20	117	111	116	111
query21	205	105	102	102
query22	4633	4431	4397	4397
query23	34836	34195	34167	34167
query24	11226	2798	2803	2798
query25	695	397	401	397
query26	1660	158	159	158
query27	2801	285	281	281
query28	8148	2509	2451	2451
query29	1001	419	422	419
query30	326	157	167	157
query31	1026	812	830	812
query32	95	56	57	56
query33	778	272	263	263
query34	1013	517	531	517
query35	896	731	756	731
query36	1088	949	969	949
query37	140	76	78	76
query38	4473	4205	4362	4205
query39	1518	1451	1459	1451
query40	292	101	101	101
query41	48	46	48	46
query42	111	100	101	100
query43	520	503	481	481
query44	1249	815	817	815
query45	186	166	167	166
query46	1157	689	705	689
query47	1947	1844	1861	1844
query48	429	324	320	320
query49	1263	418	404	404
query50	816	382	399	382
query51	7341	7190	7176	7176
query52	103	89	89	89
query53	259	182	183	182
query54	1299	444	417	417
query55	75	79	79	79
query56	275	251	259	251
query57	1284	1182	1196	1182
query58	246	210	206	206
query59	3328	3065	3073	3065
query60	342	248	236	236
query61	103	95	102	95
query62	852	681	673	673
query63	213	191	183	183
query64	5137	641	601	601
query65	3300	3226	3236	3226
query66	1434	317	323	317
query67	16150	15770	15933	15770
query68	4785	573	565	565
query69	430	257	245	245
query70	1182	1031	1086	1031
query71	333	256	245	245
query72	6397	4003	3963	3963
query73	777	358	369	358
query74	10300	9021	9085	9021
query75	3416	2662	2692	2662
query76	2857	1068	1081	1068
query77	440	267	286	267
query78	10486	9414	9327	9327
query79	1281	585	600	585
query80	1021	449	423	423
query81	538	237	239	237
query82	1208	120	119	119
query83	207	138	138	138
query84	225	72	70	70
query85	1263	304	278	278
query86	371	258	309	258
query87	4791	4644	4740	4644
query88	3547	2237	2193	2193
query89	393	291	286	286
query90	1924	187	183	183
query91	130	97	98	97
query92	58	49	50	49
query93	1080	565	554	554
query94	912	298	304	298
query95	336	249	250	249
query96	616	278	279	278
query97	2870	2784	2732	2732
query98	215	202	197	197
query99	1550	1326	1289	1289
Total cold run time: 302141 ms
Total hot run time: 192506 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.85 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6870801f4a0356ee06983f6a36c7477a28975d12, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.06	0.06
query4	1.65	0.11	0.09
query5	0.42	0.41	0.40
query6	1.14	0.64	0.66
query7	0.01	0.02	0.02
query8	0.04	0.03	0.03
query9	0.55	0.50	0.51
query10	0.57	0.55	0.56
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.63	0.60	0.59
query14	2.69	2.78	2.72
query15	0.90	0.82	0.82
query16	0.39	0.38	0.38
query17	1.04	1.08	0.98
query18	0.24	0.22	0.21
query19	1.93	1.83	1.80
query20	0.02	0.02	0.01
query21	15.36	0.60	0.58
query22	2.73	2.99	1.74
query23	17.07	0.84	0.73
query24	3.07	1.34	0.55
query25	0.28	0.09	0.11
query26	0.51	0.15	0.14
query27	0.05	0.05	0.05
query28	11.03	1.10	1.07
query29	12.56	3.24	3.22
query30	0.24	0.06	0.06
query31	2.85	0.38	0.38
query32	3.28	0.46	0.45
query33	3.02	3.09	3.01
query34	17.03	4.46	4.50
query35	4.48	4.49	4.47
query36	0.67	0.50	0.49
query37	0.08	0.06	0.05
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.51 s
Total hot run time: 31.85 s

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41463 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca6fa5f627607045b84c541b401f6c5c429270bb, data reload: false

------ Round 1 ----------------------------------
q1	17589	7489	7290	7290
q2	2047	169	180	169
q3	10686	1083	1147	1083
q4	10551	859	830	830
q5	7742	3100	3061	3061
q6	232	153	148	148
q7	1014	606	621	606
q8	9366	1972	2040	1972
q9	6588	6446	6444	6444
q10	7066	2462	2450	2450
q11	479	254	259	254
q12	415	209	208	208
q13	17770	2973	3028	2973
q14	237	204	202	202
q15	577	518	516	516
q16	683	593	573	573
q17	982	607	556	556
q18	7265	6781	6788	6781
q19	1400	1025	999	999
q20	489	181	188	181
q21	3969	3251	3154	3154
q22	1131	1013	1037	1013
Total cold run time: 108278 ms
Total hot run time: 41463 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7339	7303	7292	7292
q2	323	231	221	221
q3	3035	2964	2929	2929
q4	2111	1884	1771	1771
q5	5759	5739	5803	5739
q6	224	142	148	142
q7	2223	1873	1781	1781
q8	3373	3522	3415	3415
q9	8941	8931	8886	8886
q10	3588	3538	3558	3538
q11	595	501	493	493
q12	799	610	645	610
q13	9858	3129	3184	3129
q14	306	280	285	280
q15	563	513	523	513
q16	680	645	661	645
q17	1898	1609	1656	1609
q18	8305	7778	7557	7557
q19	1709	1577	1514	1514
q20	2117	1894	1876	1876
q21	5554	5185	5283	5185
q22	1161	1055	1063	1055
Total cold run time: 70461 ms
Total hot run time: 60180 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195198 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca6fa5f627607045b84c541b401f6c5c429270bb, data reload: false

query1	1250	945	903	903
query2	6226	2047	1981	1981
query3	10814	4051	3957	3957
query4	68032	29059	23502	23502
query5	4894	445	428	428
query6	392	164	167	164
query7	5544	289	290	289
query8	316	250	233	233
query9	8711	2744	2687	2687
query10	445	269	245	245
query11	17558	15255	15810	15255
query12	150	102	106	102
query13	1510	440	424	424
query14	10694	7104	6754	6754
query15	209	192	196	192
query16	7070	453	442	442
query17	1034	557	564	557
query18	1832	306	292	292
query19	198	155	151	151
query20	122	111	118	111
query21	203	101	98	98
query22	4400	4311	4441	4311
query23	34355	34047	33835	33835
query24	5924	2808	2717	2717
query25	523	403	395	395
query26	657	157	153	153
query27	1673	283	298	283
query28	4175	2479	2439	2439
query29	683	427	422	422
query30	228	154	151	151
query31	988	818	802	802
query32	66	54	57	54
query33	463	270	270	270
query34	911	513	504	504
query35	876	763	724	724
query36	1068	960	971	960
query37	122	72	76	72
query38	4353	4202	4196	4196
query39	1468	1550	1368	1368
query40	202	99	101	99
query41	48	53	49	49
query42	104	97	100	97
query43	512	487	481	481
query44	1156	828	837	828
query45	184	168	165	165
query46	1110	691	690	690
query47	1956	1858	1887	1858
query48	422	319	341	319
query49	754	400	408	400
query50	807	398	386	386
query51	7219	7269	7169	7169
query52	99	89	87	87
query53	249	177	182	177
query54	519	408	400	400
query55	76	96	72	72
query56	248	233	232	232
query57	1313	1192	1147	1147
query58	206	214	197	197
query59	3061	2974	2980	2974
query60	260	251	251	251
query61	105	102	103	102
query62	801	677	673	673
query63	219	187	179	179
query64	1367	629	624	624
query65	3280	3170	3198	3170
query66	736	302	303	302
query67	15940	15897	15575	15575
query68	3754	596	564	564
query69	411	250	249	249
query70	1176	1135	1136	1135
query71	351	247	242	242
query72	6344	4015	3986	3986
query73	747	359	368	359
query74	9985	8984	9002	8984
query75	3383	2647	2666	2647
query76	1736	1137	1135	1135
query77	467	277	286	277
query78	10403	9431	9396	9396
query79	1101	592	586	586
query80	695	419	444	419
query81	526	238	242	238
query82	214	114	121	114
query83	154	146	140	140
query84	276	70	75	70
query85	842	312	292	292
query86	325	259	297	259
query87	4814	4722	4693	4693
query88	3867	2187	2162	2162
query89	410	290	282	282
query90	2096	182	183	182
query91	147	102	104	102
query92	58	49	47	47
query93	1123	533	546	533
query94	789	282	277	277
query95	351	246	243	243
query96	611	278	295	278
query97	2844	2682	2711	2682
query98	212	198	195	195
query99	1599	1304	1309	1304
Total cold run time: 314901 ms
Total hot run time: 195198 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.74 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca6fa5f627607045b84c541b401f6c5c429270bb, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.41	0.40	0.41
query6	1.17	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.49	0.49
query10	0.55	0.56	0.56
query11	0.14	0.10	0.11
query12	0.13	0.10	0.11
query13	0.60	0.59	0.59
query14	2.68	2.75	2.85
query15	0.90	0.83	0.81
query16	0.37	0.38	0.37
query17	1.03	1.03	1.04
query18	0.21	0.20	0.19
query19	1.88	1.81	1.92
query20	0.01	0.01	0.02
query21	15.35	0.59	0.58
query22	2.66	2.66	1.70
query23	16.93	0.86	0.76
query24	3.47	1.74	1.40
query25	0.34	0.10	0.04
query26	0.58	0.14	0.14
query27	0.05	0.05	0.05
query28	9.75	1.10	1.09
query29	12.55	3.28	3.24
query30	0.25	0.06	0.06
query31	2.88	0.39	0.38
query32	3.26	0.46	0.47
query33	2.98	2.96	3.08
query34	16.84	4.49	4.49
query35	4.47	4.50	4.50
query36	0.67	0.50	0.49
query37	0.09	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.17 s
Total hot run time: 32.74 s

@zhangstar333
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41220 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d7ffac4eabab49bbb6c14827aec1d646dcd6e0b8, data reload: false

------ Round 1 ----------------------------------
q1	17608	7462	7235	7235
q2	2066	168	157	157
q3	10611	1074	1160	1074
q4	10528	823	792	792
q5	7763	3066	3033	3033
q6	233	151	153	151
q7	1002	608	622	608
q8	9346	1957	1989	1957
q9	6600	6456	6504	6456
q10	7100	2410	2446	2410
q11	470	272	252	252
q12	414	225	214	214
q13	17777	2993	3024	2993
q14	233	212	211	211
q15	571	523	520	520
q16	673	603	600	600
q17	983	584	541	541
q18	7310	6809	6671	6671
q19	1327	1099	1051	1051
q20	478	181	184	181
q21	3975	3143	3116	3116
q22	1128	998	997	997
Total cold run time: 108196 ms
Total hot run time: 41220 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7319	7230	7286	7230
q2	320	232	227	227
q3	2975	2916	2959	2916
q4	2063	1836	1832	1832
q5	5756	5755	5781	5755
q6	225	146	148	146
q7	2222	1863	1825	1825
q8	3364	3552	3407	3407
q9	8882	8942	8916	8916
q10	3577	3549	3584	3549
q11	612	495	508	495
q12	833	692	601	601
q13	8565	3210	3253	3210
q14	303	264	266	264
q15	582	523	516	516
q16	681	661	652	652
q17	1843	1654	1615	1615
q18	8412	7801	7649	7649
q19	1724	1694	1527	1527
q20	2080	1891	1864	1864
q21	5521	5417	5392	5392
q22	1166	1083	1043	1043
Total cold run time: 69025 ms
Total hot run time: 60631 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195594 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d7ffac4eabab49bbb6c14827aec1d646dcd6e0b8, data reload: false

query1	1237	922	937	922
query2	6226	2119	2019	2019
query3	10813	3939	3989	3939
query4	68054	29781	23626	23626
query5	4802	450	461	450
query6	398	178	171	171
query7	5607	288	294	288
query8	318	228	223	223
query9	9315	2666	2657	2657
query10	459	259	248	248
query11	17265	15331	15862	15331
query12	159	98	102	98
query13	1691	434	430	430
query14	9964	6769	7283	6769
query15	200	189	194	189
query16	7088	401	453	401
query17	1029	568	577	568
query18	1809	305	293	293
query19	218	156	151	151
query20	117	112	113	112
query21	206	100	104	100
query22	4352	4169	4354	4169
query23	34710	33860	34148	33860
query24	6104	2798	2772	2772
query25	514	395	404	395
query26	642	166	160	160
query27	1746	289	288	288
query28	3965	2450	2406	2406
query29	697	429	428	428
query30	239	158	155	155
query31	990	799	817	799
query32	68	57	58	57
query33	483	269	280	269
query34	902	513	520	513
query35	861	760	736	736
query36	1067	950	976	950
query37	120	82	74	74
query38	4415	4297	4300	4297
query39	1485	1412	1411	1411
query40	209	103	99	99
query41	48	46	46	46
query42	110	98	102	98
query43	517	498	490	490
query44	1208	825	827	825
query45	178	163	172	163
query46	1142	722	690	690
query47	1933	1817	1871	1817
query48	422	332	338	332
query49	745	420	403	403
query50	816	410	397	397
query51	7308	7195	7102	7102
query52	102	93	89	89
query53	260	179	185	179
query54	528	413	407	407
query55	76	95	75	75
query56	252	245	237	237
query57	1280	1140	1119	1119
query58	208	204	212	204
query59	3392	2957	2998	2957
query60	277	248	237	237
query61	108	124	106	106
query62	797	693	687	687
query63	223	197	176	176
query64	1346	640	615	615
query65	3271	3207	3226	3207
query66	712	297	336	297
query67	15997	15749	15663	15663
query68	3008	585	601	585
query69	422	260	252	252
query70	1197	1102	1122	1102
query71	351	262	267	262
query72	6225	3992	3966	3966
query73	758	360	361	360
query74	10029	8958	9089	8958
query75	3404	2677	2669	2669
query76	1754	1078	1063	1063
query77	487	302	272	272
query78	10541	9433	9425	9425
query79	1414	606	605	605
query80	901	441	436	436
query81	531	237	243	237
query82	1357	116	112	112
query83	225	137	153	137
query84	281	67	72	67
query85	889	305	306	305
query86	339	280	299	280
query87	4860	4716	4674	4674
query88	3573	2224	2174	2174
query89	414	292	290	290
query90	2044	186	186	186
query91	133	103	108	103
query92	66	49	51	49
query93	1860	551	557	551
query94	881	302	290	290
query95	357	248	244	244
query96	622	275	277	275
query97	2895	2718	2718	2718
query98	212	199	205	199
query99	1668	1307	1308	1307
Total cold run time: 317498 ms
Total hot run time: 195594 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.15 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d7ffac4eabab49bbb6c14827aec1d646dcd6e0b8, data reload: false

query1	0.03	0.03	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.07
query4	1.65	0.10	0.10
query5	0.43	0.41	0.40
query6	1.17	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.55	0.48	0.46
query10	0.54	0.55	0.53
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.60
query14	2.81	2.75	2.72
query15	0.91	0.83	0.83
query16	0.40	0.39	0.38
query17	1.02	1.03	1.02
query18	0.21	0.20	0.20
query19	1.86	1.86	1.78
query20	0.02	0.01	0.01
query21	15.35	0.59	0.58
query22	3.02	2.43	1.65
query23	17.06	0.99	0.86
query24	2.86	0.88	0.94
query25	0.29	0.22	0.05
query26	0.38	0.13	0.13
query27	0.05	0.05	0.04
query28	11.15	1.09	1.06
query29	12.51	3.27	3.32
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.46	0.45
query33	2.97	3.02	3.02
query34	16.97	4.53	4.43
query35	4.55	4.49	4.46
query36	0.66	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.03	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.52 s
Total hot run time: 32.15 s

liaoxin01
liaoxin01 approved these changes Nov 4, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 4, 2024

PR approved by at least one committer and no changes requested.

@KassieZ KassieZ mentioned this pull request Nov 5, 2024
Closed
HappenLee
HappenLee approved these changes Nov 6, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit 6cfecba into apache:master Nov 6, 2024
26 of 28 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 6, 2024
@zhangstar333
[improve](udf) support load data with udf functions (#43029)
e4487e3 
1. What problem was fixed (it's best to include specific error reporting
information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what
is it now, why was it modified, and what possible impacts might there
be.
3. What features were added. Why this function was added.
4. Which codes were refactored and why this part of the code was
refactored.
5. Which functions were optimized and what is the difference before and
after the optimization.
@github-actions github-actions bot mentioned this pull request Nov 6, 2024
Merged
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
@github-actions @zhangstar333
branch-3.0: [improve](udf) support load data with udf functions (#43361)
6d83909 
Cherry-picked from #43029

Co-authored-by: zhangstar333 <[email protected]>
@gavinchou gavinchou added cir and removed approved Indicates a PR has been approved by one committer. labels Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@liaoxin01 liaoxin01 liaoxin01 approved these changes

@HappenLee HappenLee HappenLee approved these changes

@dataroaring dataroaring dataroaring left review comments

Assignees
No one assigned
Labels
cir dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zhangstar333 @doris-robot @HappenLee @liaoxin01 @dataroaring @gavinchou