how to improve the performance of group by with tens of billions of data use trino

[yx-keith]

[jiangzhx]

This depends on many possibilities.
for example:

1. group by on high cardinality or low cardinality.
2. group by key already bucketed or not.

[jiangzhx]

1.Try to bucket group by key "name". The bucket number should be greater than the number of worker nodes (7).
Doing this will provide benefits from the "bucket_execution_enabled" feature, which will reduce the cost of "Internal Network Data" .

2.Sort data with the column "name".
Regarding this, I am not 100% sure. Does Trino have the ability of "streaming aggregation"? It will reduce memory usage.

3.does column "dt" was partition column?

4.how about file size in hdfs?

[yx-keith]

does bucket group by key "name" means to bucket table when create hive table?

yes, dt is partition column.

[jiangzhx]

does bucket group by key "name" means to bucket table when create hive table?

yes, dt is partition column.

yes

[yx-keith]

file size is 3.4TB , number of files is 3728

[yx-keith]

most of time(80%) is spent in source stage, wall time is 1.73 times of cpu time in source stage, does it mean that most of time is spent on I/O?