Introduce operator of window function. #14213

Sh-Zh-7 · 2024-11-26T18:05:32Z

This PR introduces a window operator in IoTDB, which can be used as backend of window function in SQL.

...he/iotdb/db/queryengine/execution/operator/source/relational/aggregation/SumAccumulator.java

...che/iotdb/db/queryengine/execution/operator/process/window/exception/FrameTypeException.java

.../iotdb/db/queryengine/execution/operator/process/window/function/value/LeadFunctionTest.java

Beyyes · 2025-01-13T02:30:33Z

...db/db/queryengine/execution/operator/process/window/function/aggregate/WindowAggregator.java

+    accumulator.evaluateFinal(columnBuilder);
+  }
+
+  public void processStatistics(Statistics[] statistics) {


Can WindowAggregator use statistics?

Theoretically yes, but it's really rare. A frame must not be smaller than a chunk/page/tsfile so that it can use its statistics.

So I choose to remain this method.

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/utils/RowComparator.java

Beyyes · 2025-01-13T02:56:54Z

...java/org/apache/iotdb/db/queryengine/execution/operator/process/window/utils/ColumnList.java

+  }
+
+  public boolean getBoolean(int position) {
+    ColumnListIndex columnListIndex = getColumnIndex(position);


Suggested change

ColumnListIndex columnListIndex = getColumnIndex(position);

ColumnListIndex columnListIndex = getColumnIndex(position);

int columnIndex = columnListIndex.getColumnIndex();

These two lines are duplicated in getXXX method

These two getColumnIndex are different methods.

Maybe getColumnListIndex is more intuitive.

...java/org/apache/iotdb/db/queryengine/execution/operator/process/window/utils/ColumnList.java

...apache/iotdb/db/queryengine/execution/operator/process/window/partition/frame/RowsFrame.java

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

...pache/iotdb/db/queryengine/execution/operator/process/window/partition/frame/RangeFrame.java

Beyyes · 2025-01-13T06:37:45Z

...apache/iotdb/db/queryengine/execution/operator/process/window/partition/frame/FrameInfo.java

+  // Then it is stored in double
+  private final double startOffset; // For PRECEDING and FOLLOWING use
+  private final FrameBoundType endType;
+  private final double endOffset; // Same as startOffset


I see all types of xxxOffset are int, if it's needed to use double?

RangeFrame may have offset with decimal, it cannot be stored by int type.

And the FrameInfo instances count is equal to the number of frame clause in user's SQL, so it wont cause too much extra memory footprint.

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/partition/Partition.java

iotdb-core/confignode/src/test/resources/confignode1conf/iotdb-system.properties

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

Beyyes · 2025-01-20T09:04:17Z

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

+
+      // May return null if builder is not full
+      return transform();
+    } else if (!cachedTsBlocks.isEmpty()) {


consider memory control of cachedBlocks

Done. Support for explain analysis is added as well.

...db/db/queryengine/execution/operator/process/window/function/aggregate/WindowAggregator.java

…ons.

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

Beyyes · 2025-01-24T09:49:21Z

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

+    // PartitionExecutor only hold reference to TsBlock
+    // So only cached TsBlocks are considered
+    long maxPeekMemoryFromCurrent =
+        (long) cachedTsBlocks.size()


Method calculateMaxPeekMemory is used before the execution of operator, so cachedTsBlocks.size() will always equals to 0.
And the memory of cachedTsBlocks haven been managed by memoryReservationManager.

Beyyes · 2025-01-24T09:52:00Z

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

+    long reserved = tsBlock.getTotalInstanceSize();
+    memoryReservationManager.reserveMemoryCumulatively(reserved);
+    totalMemorySize += reserved;
+    operatorContext.recordSpecifiedInfo(CURRENT_USED_MEMORY, Long.toString(totalMemorySize));


Maybe it's enough for only recording the max_used_memory.

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java

Sh-Zh-7 added 17 commits November 13, 2024 00:16

Add basic work processors.

65d341c

Rewrite all tests.

e8c2a14

Refactor all files.

121439f

Rename package from 'processor' to 'workers'

6366d71

Decrease file's number.

e558834

Add license header to all files.

0eca37a

Add necessary comment to all files.

54571b2

Include files borrowed from Trino.

b4741a1

Remove all requireNonNulls.

80fa918

Format all files using spotless plugin.

b79a557

Make ctor in WorkProcessor public.

b68f2c7

Update WorkProcessors.

b1d7db8

Window op prototype.

55684e8

Merge master branch from origin.

465bce7

Mvn spotless apply.

922d3c3

Very, very simple prototype.

fe21202

Run mvn spotless apply.

d5e5622

Sh-Zh-7 marked this pull request as draft November 27, 2024 01:25

Sh-Zh-7 added 12 commits November 29, 2024 00:42

Support multi-channel partition and can sort successfully now.

e05b79d

Finish three types of frame.

6cb7d00

Refactor whole range module.

488a3ae

Adjust project structure.

3839bdf

Finish writing value window functions.

4849895

Finish writing rank window functions.

a7db416

Shrink exception implementation in window function.

af656e2

Combine frame and partition directory.

631a367

Run mvn spotless apply.

7d4ae3c

Support multiple window functions and frames.

ddf84cd

Temporarily save my workspace.

ca9b93c

Add removeInput and hasRemoveInput method to specific Accumulators.

c937900

Sh-Zh-7 added 2 commits December 23, 2024 22:53

Introduce timestamp datatype in range frame.

5d376ae

Perhaps last mvn spotless::apply before code review.

28a58ad

Beyyes self-requested a review January 2, 2025 09:14

Beyyes reviewed Jan 13, 2025

View reviewed changes

...a/org/apache/iotdb/db/queryengine/execution/operator/process/window/TableWindowOperator.java Outdated Show resolved Hide resolved

Sh-Zh-7 added 3 commits January 15, 2025 02:10

Little modification introduced by code review.

1b2270d

First mvn spotless::apply after code review.

b2ef78b

Resolve conflicts

e406ebd

Beyyes reviewed Jan 15, 2025

View reviewed changes

Beyyes approved these changes Jan 20, 2025

View reviewed changes

iotdb-core/confignode/src/test/resources/confignode1conf/iotdb-system.properties Outdated Show resolved Hide resolved

Beyyes requested changes Jan 20, 2025

View reviewed changes

Beyyes reviewed Jan 21, 2025

View reviewed changes

...db/db/queryengine/execution/operator/process/window/function/aggregate/WindowAggregator.java Outdated Show resolved Hide resolved

Sh-Zh-7 added 11 commits January 22, 2025 00:36

Modify by code review(PART I).

5782043

Merge branch 'master' into window_op

f7471f3

Modify by code review(PART II).

6b07f8e

mvn spotless::apply again.

75d0454

Modify code by code review(PART III).

5f42807

Extract common part from value window function and rank window functi…

c5bc897

…ons.

Perhaps last mvn spotless::apply.

a4c97ab

Fix unit tests errors.

ab111ad

Temporary commit for refactoring.

be505c3

Support offset in columns.

d8381b4

Maybe last format before New Year.

c12995e

Beyyes reviewed Jan 24, 2025

View reviewed changes

Sh-Zh-7 added 3 commits January 24, 2025 19:04

Add license header to each files.

18d240d

Modify code by code review(PART IV).

da891c9

Maybe last code format.

2422d2e

Sh-Zh-7 requested a review from Beyyes January 24, 2025 11:16

Beyyes approved these changes Jan 24, 2025

View reviewed changes

Beyyes merged commit f2d24ff into apache:master Jan 24, 2025
34 of 35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce operator of window function. #14213

Introduce operator of window function. #14213

Sh-Zh-7 commented Nov 26, 2024 •

edited

Loading

Beyyes Jan 13, 2025

Sh-Zh-7 Jan 14, 2025

Beyyes Jan 13, 2025

Sh-Zh-7 Jan 14, 2025

Sh-Zh-7 Jan 14, 2025

Beyyes Jan 13, 2025

Sh-Zh-7 Jan 21, 2025

Beyyes Jan 20, 2025

Sh-Zh-7 Jan 21, 2025

Beyyes Jan 24, 2025

Sh-Zh-7 Jan 24, 2025

Beyyes Jan 24, 2025

Sh-Zh-7 Jan 24, 2025

	ColumnListIndex columnListIndex = getColumnIndex(position);
	ColumnListIndex columnListIndex = getColumnIndex(position);
	int columnIndex = columnListIndex.getColumnIndex();

Introduce operator of window function. #14213

Introduce operator of window function. #14213

Conversation

Sh-Zh-7 commented Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Sh-Zh-7 commented Nov 26, 2024 •

edited

Loading