Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement](function)make SUBSTRING_INDEX function DEPEND_ON_ARGUMENT #30392

Merged
merged 35 commits into from
Feb 2, 2024

Conversation

koarz
Copy link
Contributor

@koarz koarz commented Jan 25, 2024

Proposed changes

Issue Number: #27435

make function SUBSTRING_INDEX DEPEND_ON_ARGUMENT

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@koarz
Copy link
Contributor Author

koarz commented Jan 26, 2024

run buildall

@koarz koarz changed the title [fix](function)make SUBSTRING_INDEX function DEPEND_ON_ARGUMENT [enhancement](function)make SUBSTRING_INDEX function DEPEND_ON_ARGUMENT Jan 26, 2024
@koarz koarz changed the title [enhancement](function)make SUBSTRING_INDEX function DEPEND_ON_ARGUMENT [Enhancement](function)make SUBSTRING_INDEX function DEPEND_ON_ARGUMENT Jan 26, 2024
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@doris-robot
Copy link

TPC-H: Total hot run time: 38626 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7ecc51d500aaa808e669a22b2add2d0281bf8e2, data reload: false

------ Round 1 ----------------------------------
q1	18632	5372	5536	5372
q2	2305	141	135	135
q3	11325	1168	1155	1155
q4	10734	752	772	752
q5	7751	3148	3176	3148
q6	203	121	124	121
q7	854	496	484	484
q8	9214	1887	1939	1887
q9	7198	6334	6349	6334
q10	8224	3020	3040	3020
q11	426	197	203	197
q12	362	188	191	188
q13	17988	3371	3346	3346
q14	245	217	211	211
q15	553	509	528	509
q16	431	376	382	376
q17	952	552	489	489
q18	7599	6987	6668	6668
q19	1553	1306	1290	1290
q20	597	308	284	284
q21	2799	2358	2407	2358
q22	362	302	310	302
Total cold run time: 110307 ms
Total hot run time: 38626 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5451	5229	5205	5205
q2	329	218	210	210
q3	3332	3220	3215	3215
q4	2075	2035	2046	2035
q5	5948	5884	6053	5884
q6	195	117	120	117
q7	2358	1918	1897	1897
q8	3199	3369	3377	3369
q9	9009	8700	8755	8700
q10	3890	3778	3887	3778
q11	550	432	461	432
q12	790	637	599	599
q13	16922	3190	3137	3137
q14	284	259	260	259
q15	561	504	512	504
q16	509	457	449	449
q17	1861	1832	1864	1832
q18	9674	20246	9653	9653
q19	23898	1543	1516	1516
q20	4574	1929	1919	1919
q21	14583	5426	5530	5426
q22	952	534	590	534
Total cold run time: 110944 ms
Total hot run time: 60670 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187356 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7ecc51d500aaa808e669a22b2add2d0281bf8e2, data reload: false

query1	940	339	325	325
query2	6554	2031	1977	1977
query3	6696	207	209	207
query4	31992	22244	22292	22244
query5	4458	444	452	444
query6	251	156	154	154
query7	4607	270	265	265
query8	228	180	184	180
query9	8392	2644	2636	2636
query10	432	233	261	233
query11	17173	15522	15594	15522
query12	126	71	70	70
query13	1686	388	379	379
query14	10660	6964	7054	6964
query15	209	179	185	179
query16	5802	254	252	252
query17	955	474	480	474
query18	1782	265	245	245
query19	173	137	133	133
query20	74	76	70	70
query21	188	132	129	129
query22	5033	4913	4944	4913
query23	31742	30867	30832	30832
query24	12522	2800	2774	2774
query25	569	324	315	315
query26	1770	145	144	144
query27	3220	288	288	288
query28	6917	1861	1848	1848
query29	1763	607	618	607
query30	276	136	134	134
query31	927	762	762	762
query32	89	49	50	49
query33	700	217	211	211
query34	1142	454	474	454
query35	879	774	768	768
query36	1361	1228	1214	1214
query37	91	57	60	57
query38	3389	3325	3196	3196
query39	1309	1295	1260	1260
query40	343	89	81	81
query41	38	36	35	35
query42	91	84	84	84
query43	529	462	448	448
query44	1096	694	705	694
query45	195	180	173	173
query46	1072	647	647	647
query47	1708	1608	1572	1572
query48	385	300	304	300
query49	1176	297	295	295
query50	667	313	306	306
query51	5277	5178	5197	5178
query52	93	71	75	71
query53	321	264	260	260
query54	240	180	181	180
query55	79	79	79	79
query56	186	170	168	168
query57	1013	915	922	915
query58	193	157	164	157
query59	2968	2706	2750	2706
query60	208	185	187	185
query61	92	83	85	83
query62	632	366	373	366
query63	272	270	258	258
query64	6156	1768	1754	1754
query65	3398	3274	3232	3232
query66	1265	316	316	316
query67	15637	15126	15317	15126
query68	11029	520	541	520
query69	596	309	282	282
query70	1608	1543	1509	1509
query71	10419	10198	10202	10198
query72	4835	2857	2845	2845
query73	2135	315	314	314
query74	6974	6411	6468	6411
query75	4576	2348	2269	2269
query76	6283	1041	1014	1014
query77	665	235	233	233
query78	9162	8923	8615	8615
query79	1022	494	517	494
query80	539	324	317	317
query81	450	202	202	202
query82	176	78	78	78
query83	137	129	121	121
query84	267	72	71	71
query85	1064	334	346	334
query86	391	384	367	367
query87	3504	3357	3297	3297
query88	3006	2226	2202	2202
query89	437	359	348	348
query90	1950	189	187	187
query91	154	123	132	123
query92	50	43	43	43
query93	2230	430	442	430
query94	1237	160	157	157
query95	503	459	456	456
query96	616	320	331	320
query97	4275	4131	4130	4130
query98	205	191	188	188
query99	1001	705	704	704
Total cold run time: 303903 ms
Total hot run time: 187356 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7ecc51d500aaa808e669a22b2add2d0281bf8e2, data reload: false

query1	0.04	0.02	0.02
query2	0.06	0.03	0.02
query3	0.22	0.05	0.04
query4	1.71	0.07	0.07
query5	0.54	0.52	0.52
query6	1.30	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.51	0.50
query10	0.56	0.56	0.56
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.61	0.62	0.61
query14	0.81	0.81	0.81
query15	0.78	0.78	0.77
query16	0.41	0.39	0.39
query17	1.01	0.97	1.00
query18	0.23	0.28	0.25
query19	1.87	1.76	1.77
query20	0.02	0.01	0.01
query21	15.41	0.58	0.57
query22	2.29	1.89	2.40
query23	17.40	0.82	0.81
query24	2.89	1.12	1.03
query25	0.42	0.16	0.22
query26	0.57	0.13	0.13
query27	0.05	0.06	0.05
query28	10.92	0.78	0.77
query29	12.55	3.19	3.24
query30	0.52	0.47	0.48
query31	2.78	0.36	0.36
query32	3.36	0.48	0.49
query33	3.25	3.23	3.22
query34	16.23	4.27	4.25
query35	4.31	4.28	4.25
query36	1.10	1.08	1.07
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.02	0.02	0.01
Total cold run time: 105.48 s
Total hot run time: 31.11 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b7ecc51d500aaa808e669a22b2add2d0281bf8e2 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

@koarz
Copy link
Contributor Author

koarz commented Jan 26, 2024

run p0

Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@koarz
Copy link
Contributor Author

koarz commented Jan 31, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}

bool use_default_implementation_for_nulls() const override { return true; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute_impl' has cognitive complexity of 86 (threshold 50) [readability-function-cognitive-complexity]

    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
           ^
Additional context

be/src/vec/functions/function_string.h:1888: +1, including nesting penalty of 0, nesting level increased to 1

        if (part_number == 0 || delimiter_size == 0) {
        ^

be/src/vec/functions/function_string.h:1888: +1

        if (part_number == 0 || delimiter_size == 0) {
                             ^

be/src/vec/functions/function_string.h:1889: +2, including nesting penalty of 1, nesting level increased to 2

            for (size_t i = 0; i < input_rows_count; ++i) {
            ^

be/src/vec/functions/function_string.h:1892: +1, nesting level increased to 1

        } else if (part_number > 0) {
               ^

be/src/vec/functions/function_string.h:1893: +2, including nesting penalty of 1, nesting level increased to 2

            if (delimiter_size == 1) {
            ^

be/src/vec/functions/function_string.h:1895: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < input_rows_count; ++i) {
                ^

be/src/vec/functions/function_string.h:1899: +4, including nesting penalty of 3, nesting level increased to 4

                    while (num < part_number) {
                    ^

be/src/vec/functions/function_string.h:1903: +5, including nesting penalty of 4, nesting level increased to 5

                        if (pos != nullptr) {
                        ^

be/src/vec/functions/function_string.h:1906: +1, nesting level increased to 5

                        } else {
                          ^

be/src/vec/functions/function_string.h:1908: +6, including nesting penalty of 5, nesting level increased to 6

                            num = (num == 0) ? 0 : num + 1;
                                             ^

be/src/vec/functions/function_string.h:1913: +4, including nesting penalty of 3, nesting level increased to 4

                    if (num == part_number) {
                    ^

be/src/vec/functions/function_string.h:1918: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:1923: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/functions/function_string.h:1926: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < input_rows_count; ++i) {
                ^

be/src/vec/functions/function_string.h:1930: +4, including nesting penalty of 3, nesting level increased to 4

                    while (num < part_number) {
                    ^

be/src/vec/functions/function_string.h:1934: +5, including nesting penalty of 4, nesting level increased to 5

                        if (pos < str.data + str.size) {
                        ^

be/src/vec/functions/function_string.h:1937: +1, nesting level increased to 5

                        } else {
                          ^

be/src/vec/functions/function_string.h:1939: +6, including nesting penalty of 5, nesting level increased to 6

                            num = (num == 0) ? 0 : num + 1;
                                             ^

be/src/vec/functions/function_string.h:1944: +4, including nesting penalty of 3, nesting level increased to 4

                    if (num == part_number) {
                    ^

be/src/vec/functions/function_string.h:1949: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:1955: +1, nesting level increased to 1

        } else {
          ^

be/src/vec/functions/function_string.h:1958: +2, including nesting penalty of 1, nesting level increased to 2

            for (size_t i = 0; i < input_rows_count; ++i) {
            ^

be/src/vec/functions/function_string.h:1965: +3, including nesting penalty of 2, nesting level increased to 3

                while (num <= part_number && offset >= 0) {
                ^

be/src/vec/functions/function_string.h:1965: +1

                while (num <= part_number && offset >= 0) {
                                          ^

be/src/vec/functions/function_string.h:1967: +4, including nesting penalty of 3, nesting level increased to 4

                    if (offset != -1) {
                    ^

be/src/vec/functions/function_string.h:1968: +5, including nesting penalty of 4, nesting level increased to 5

                        if (++num == part_number) {
                        ^

be/src/vec/functions/function_string.h:1974: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:1978: +3, including nesting penalty of 2, nesting level increased to 3

                num = (offset == -1 && num != 0) ? num + 1 : num;
                                                 ^

be/src/vec/functions/function_string.h:1978: +1

                num = (offset == -1 && num != 0) ? num + 1 : num;
                                    ^

be/src/vec/functions/function_string.h:1980: +3, including nesting penalty of 2, nesting level increased to 3

                if (num == part_number) {
                ^

be/src/vec/functions/function_string.h:1981: +4, including nesting penalty of 3, nesting level increased to 4

                    if (offset == -1) {
                    ^

be/src/vec/functions/function_string.h:1984: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:1990: +1, nesting level increased to 3

                } else {
                  ^

}

bool use_default_implementation_for_nulls() const override { return true; }
Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute_impl' exceeds recommended size/complexity thresholds [readability-function-size]

    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
           ^
Additional context

be/src/vec/functions/function_string.h:1862: 136 lines including whitespace and comments (threshold 80)

    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
           ^

String get_name() const override { return name; }
size_t get_number_of_arguments() const override { return 3; }

DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'get_return_type_impl' can be made static [readability-convert-member-functions-to-static]

Suggested change
DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
static DataTypePtr get_return_type_impl(const DataTypes& arguments) override {

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions


DataTypePtr get_return_type_impl(const DataTypes& arguments) const override {
return make_nullable(std::make_shared<DataTypeString>());
}

Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'execute_impl' has cognitive complexity of 87 (threshold 50) [readability-function-cognitive-complexity]

    Status execute_impl(FunctionContext* context, Block& block, const ColumnNumbers& arguments,
           ^
Additional context

be/src/vec/functions/function_string.h:2029: +1, including nesting penalty of 0, nesting level increased to 1

        if (const auto* nullable = check_and_get_column<const ColumnNullable>(*content_column)) {
        ^

be/src/vec/functions/function_string.h:2048: +1, including nesting penalty of 0, nesting level increased to 1

        if (part_number == 0 || delimiter_size == 0) {
        ^

be/src/vec/functions/function_string.h:2048: +1

        if (part_number == 0 || delimiter_size == 0) {
                             ^

be/src/vec/functions/function_string.h:2049: +2, including nesting penalty of 1, nesting level increased to 2

            for (size_t i = 0; i < input_rows_count; ++i) {
            ^

be/src/vec/functions/function_string.h:2052: +1, nesting level increased to 1

        } else if (part_number > 0) {
               ^

be/src/vec/functions/function_string.h:2053: +2, including nesting penalty of 1, nesting level increased to 2

            if (delimiter_size == 1) {
            ^

be/src/vec/functions/function_string.h:2055: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < input_rows_count; ++i) {
                ^

be/src/vec/functions/function_string.h:2059: +4, including nesting penalty of 3, nesting level increased to 4

                    while (num < part_number) {
                    ^

be/src/vec/functions/function_string.h:2063: +5, including nesting penalty of 4, nesting level increased to 5

                        if (pos != nullptr) {
                        ^

be/src/vec/functions/function_string.h:2066: +1, nesting level increased to 5

                        } else {
                          ^

be/src/vec/functions/function_string.h:2068: +6, including nesting penalty of 5, nesting level increased to 6

                            num = (num == 0) ? 0 : num + 1;
                                             ^

be/src/vec/functions/function_string.h:2073: +4, including nesting penalty of 3, nesting level increased to 4

                    if (num == part_number) {
                    ^

be/src/vec/functions/function_string.h:2078: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:2083: +1, nesting level increased to 2

            } else {
              ^

be/src/vec/functions/function_string.h:2086: +3, including nesting penalty of 2, nesting level increased to 3

                for (size_t i = 0; i < input_rows_count; ++i) {
                ^

be/src/vec/functions/function_string.h:2090: +4, including nesting penalty of 3, nesting level increased to 4

                    while (num < part_number) {
                    ^

be/src/vec/functions/function_string.h:2094: +5, including nesting penalty of 4, nesting level increased to 5

                        if (pos < str.data + str.size) {
                        ^

be/src/vec/functions/function_string.h:2097: +1, nesting level increased to 5

                        } else {
                          ^

be/src/vec/functions/function_string.h:2099: +6, including nesting penalty of 5, nesting level increased to 6

                            num = (num == 0) ? 0 : num + 1;
                                             ^

be/src/vec/functions/function_string.h:2104: +4, including nesting penalty of 3, nesting level increased to 4

                    if (num == part_number) {
                    ^

be/src/vec/functions/function_string.h:2109: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:2115: +1, nesting level increased to 1

        } else {
          ^

be/src/vec/functions/function_string.h:2118: +2, including nesting penalty of 1, nesting level increased to 2

            for (size_t i = 0; i < input_rows_count; ++i) {
            ^

be/src/vec/functions/function_string.h:2125: +3, including nesting penalty of 2, nesting level increased to 3

                while (num <= part_number && offset >= 0) {
                ^

be/src/vec/functions/function_string.h:2125: +1

                while (num <= part_number && offset >= 0) {
                                          ^

be/src/vec/functions/function_string.h:2127: +4, including nesting penalty of 3, nesting level increased to 4

                    if (offset != -1) {
                    ^

be/src/vec/functions/function_string.h:2128: +5, including nesting penalty of 4, nesting level increased to 5

                        if (++num == part_number) {
                        ^

be/src/vec/functions/function_string.h:2134: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:2138: +3, including nesting penalty of 2, nesting level increased to 3

                num = (offset == -1 && num != 0) ? num + 1 : num;
                                                 ^

be/src/vec/functions/function_string.h:2138: +1

                num = (offset == -1 && num != 0) ? num + 1 : num;
                                    ^

be/src/vec/functions/function_string.h:2140: +3, including nesting penalty of 2, nesting level increased to 3

                if (num == part_number) {
                ^

be/src/vec/functions/function_string.h:2141: +4, including nesting penalty of 3, nesting level increased to 4

                    if (offset == -1) {
                    ^

be/src/vec/functions/function_string.h:2144: +1, nesting level increased to 4

                    } else {
                      ^

be/src/vec/functions/function_string.h:2150: +1, nesting level increased to 3

                } else {
                  ^

@doris-robot
Copy link

TPC-H: Total hot run time: 37247 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e19b442297b5a985377f46bbf4fd8c9e6ef528c, data reload: false

------ Round 1 ----------------------------------
q1	17990	5072	4510	4510
q2	2454	146	136	136
q3	11846	951	934	934
q4	5244	798	739	739
q5	8264	2917	2903	2903
q6	189	123	122	122
q7	1165	747	721	721
q8	9446	2016	2027	2016
q9	7278	6377	6327	6327
q10	8106	2397	2493	2397
q11	424	213	196	196
q12	775	287	271	271
q13	18009	3314	3287	3287
q14	287	255	244	244
q15	524	502	483	483
q16	477	412	403	403
q17	941	552	512	512
q18	6814	6102	5945	5945
q19	1584	1424	1501	1424
q20	578	322	332	322
q21	6946	3152	3069	3069
q22	818	286	306	286
Total cold run time: 110159 ms
Total hot run time: 37247 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4482	4493	4395	4395
q2	333	240	220	220
q3	3033	2873	2838	2838
q4	1911	1651	1758	1651
q5	5241	5291	5272	5272
q6	193	116	115	115
q7	2147	1762	1753	1753
q8	3115	3283	3270	3270
q9	8400	8341	8281	8281
q10	5979	3589	3588	3588
q11	566	446	447	446
q12	736	570	585	570
q13	6840	3111	3104	3104
q14	275	249	253	249
q15	540	491	489	489
q16	492	475	446	446
q17	1860	1671	1719	1671
q18	7979	7867	7437	7437
q19	6213	1526	1577	1526
q20	2139	1937	1906	1906
q21	4793	4576	4653	4576
q22	532	462	471	462
Total cold run time: 67799 ms
Total hot run time: 54265 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8623/23833)
Line Coverage: 28.24% (70545/249847)
Region Coverage: 27.25% (36404/133589)
Branch Coverage: 24.03% (18642/77564)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2e19b442297b5a985377f46bbf4fd8c9e6ef528c_2e19b442297b5a985377f46bbf4fd8c9e6ef528c/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172750 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e19b442297b5a985377f46bbf4fd8c9e6ef528c, data reload: false

query1	927	335	328	328
query2	6552	1859	1818	1818
query3	6718	205	195	195
query4	28089	22142	22139	22139
query5	4454	404	400	400
query6	254	173	159	159
query7	4602	263	256	256
query8	245	172	174	172
query9	9172	2340	2301	2301
query10	416	205	185	185
query11	16361	15365	15413	15365
query12	126	67	61	61
query13	1667	381	382	381
query14	9190	6584	6589	6584
query15	203	181	180	180
query16	6163	265	244	244
query17	1798	462	464	462
query18	1778	255	257	255
query19	166	133	133	133
query20	75	70	67	67
query21	193	135	128	128
query22	4949	4732	4411	4411
query23	31172	30369	30544	30369
query24	12416	2733	2727	2727
query25	608	313	304	304
query26	2112	134	134	134
query27	3262	280	290	280
query28	7731	1866	1852	1852
query29	2352	637	618	618
query30	287	136	134	134
query31	908	700	714	700
query32	91	50	49	49
query33	723	204	207	204
query34	1227	463	470	463
query35	854	774	760	760
query36	1330	1112	1148	1112
query37	176	57	57	57
query38	3267	3158	3167	3158
query39	1299	1248	1243	1243
query40	324	85	81	81
query41	38	37	35	35
query42	88	86	80	80
query43	509	478	475	475
query44	1127	701	704	701
query45	193	183	174	174
query46	1036	642	645	642
query47	1515	1448	1475	1448
query48	375	300	304	300
query49	1272	275	283	275
query50	687	314	318	314
query51	5276	5199	5108	5108
query52	87	85	72	72
query53	337	273	256	256
query54	257	177	186	177
query55	79	74	71	71
query56	183	162	159	159
query57	962	890	893	890
query58	187	147	152	147
query59	2347	2292	2361	2292
query60	198	174	175	174
query61	82	90	82	82
query62	642	360	356	356
query63	279	257	262	257
query64	5602	1735	1726	1726
query65	3277	3221	3209	3209
query66	1486	315	305	305
query67	14375	14305	14136	14136
query68	3865	503	501	501
query69	418	297	302	297
query70	1554	1538	1530	1530
query71	307	213	216	213
query72	3736	2826	2885	2826
query73	679	314	318	314
query74	6570	6236	6297	6236
query75	2847	2314	2324	2314
query76	2943	955	956	955
query77	426	227	224	224
query78	8944	8811	8539	8539
query79	2649	490	487	487
query80	1327	320	312	312
query81	523	199	200	199
query82	928	81	81	81
query83	221	117	112	112
query84	232	73	73	73
query85	1516	346	319	319
query86	451	411	385	385
query87	3431	3321	3281	3281
query88	3675	2211	2199	2199
query89	460	357	336	336
query90	2242	185	180	180
query91	153	120	135	120
query92	54	44	45	44
query93	1630	431	424	424
query94	1542	161	157	157
query95	494	448	457	448
query96	613	319	312	312
query97	4271	4426	4133	4133
query98	215	214	188	188
query99	950	683	681	681
Total cold run time: 276599 ms
Total hot run time: 172750 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e19b442297b5a985377f46bbf4fd8c9e6ef528c, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.05	0.06
query4	1.68	0.10	0.09
query5	0.52	0.51	0.51
query6	1.18	0.65	0.63
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.54	0.51	0.48
query10	0.56	0.57	0.55
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.61	0.60
query14	0.77	0.80	0.81
query15	0.80	0.77	0.77
query16	0.40	0.36	0.39
query17	1.05	1.03	1.04
query18	0.22	0.26	0.23
query19	1.90	1.79	1.79
query20	0.01	0.02	0.01
query21	15.40	0.58	0.56
query22	2.35	2.66	1.51
query23	17.18	0.94	0.76
query24	2.53	1.25	0.85
query25	0.38	0.15	0.09
query26	0.52	0.14	0.13
query27	0.06	0.05	0.05
query28	11.27	0.87	0.84
query29	12.53	3.15	3.24
query30	0.60	0.54	0.53
query31	2.78	0.35	0.36
query32	3.38	0.47	0.48
query33	3.23	3.25	3.21
query34	16.11	4.36	4.27
query35	4.33	4.32	4.36
query36	1.10	1.07	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.12
query41	0.06	0.01	0.02
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.98 s
Total hot run time: 30.56 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2e19b442297b5a985377f46bbf4fd8c9e6ef528c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@koarz koarz requested a review from HappenLee January 31, 2024 13:12
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 1, 2024
Copy link
Contributor

github-actions bot commented Feb 1, 2024

PR approved by at least one committer and no changes requested.

@HappenLee HappenLee merged commit fcfd155 into apache:master Feb 2, 2024
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants