Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Drop RowConverter from GroupOrderingPartial #14566

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

ctsk
Copy link

@ctsk ctsk commented Feb 9, 2025

Which issue does this PR close?

Rationale for this change

Faster is better?

What changes are included in this PR?

Are these changes tested?

Yes, added a new test.

Are there any user-facing changes?

No.

(The input_schema argument of GroupOrderingPartial::try_new is no longer used and can be removed, this is not included in this PR)

@github-actions github-actions bot added the physical-expr Physical Expressions label Feb 9, 2025
@ctsk ctsk force-pushed the faster-partial-groupordering branch from d7e7376 to a7324c7 Compare February 10, 2025 08:18
@ctsk
Copy link
Author

ctsk commented Feb 10, 2025

Micro-benchmark results

  • group_ordering_$n benchmarks the partial group ordering where the ordering contains $n columns. Each Columns consists of 8192 Int32s.
group_ordering_partial/order_indices_1
                        time:   [16.814 µs 17.340 µs 17.934 µs]
                        change: [-71.063% -70.318% -69.568%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_2
                        time:   [23.585 µs 23.907 µs 24.176 µs]
                        change: [-65.486% -64.711% -63.930%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_4
                        time:   [28.452 µs 28.603 µs 28.794 µs]
                        change: [-52.826% -52.404% -51.942%] (p = 0.00 < 0.05)
                        Performance has improved.
group_ordering_partial/order_indices_8
                        time:   [34.737 µs 34.828 µs 34.935 µs]
                        change: [-64.151% -63.876% -63.589%] (p = 0.00 < 0.05)
                        Performance has improved.

@ctsk ctsk force-pushed the faster-partial-groupordering branch from cccc8ea to 0d778f5 Compare February 10, 2025 08:40
@ctsk ctsk changed the title Drop RowConverter from GroupOrderingPartial perf: Drop RowConverter from GroupOrderingPartial Feb 10, 2025
@ctsk ctsk force-pushed the faster-partial-groupordering branch from 0d778f5 to ec21860 Compare February 12, 2025 16:04
@2010YOUY01
Copy link
Contributor

Thank you, this looks good to me. Let's get the CI fixed.

https://github.com/apache/datafusion/blob/main/datafusion/core/tests/fuzz_cases/aggregate_fuzz.rs should have good coverage for edge cases in group by with partial ordering, so I think this change is safe

@ctsk ctsk force-pushed the faster-partial-groupordering branch from ec21860 to 2961710 Compare February 13, 2025 11:11
@ctsk
Copy link
Author

ctsk commented Feb 13, 2025

Thank you for taking the time to review!

impl GroupOrderingPartial {
pub fn try_new(
input_schema: &Schema,
_input_schema: &Schema,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not remove this arg?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This way we can avoid API change, though I don't have a strong preference

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe we can add a comment or something as a follow on pr

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, the input_schema argument in order::GroupOrdering::try_new will also be unnecessary

impl GroupOrdering {
/// Create a `GroupOrdering` for the specified ordering
pub fn try_new(
input_schema: &Schema,
mode: &InputOrderMode,
ordering: &LexOrdering,
) -> Result<Self> {
match mode {
InputOrderMode::Linear => Ok(GroupOrdering::None),
InputOrderMode::PartiallySorted(order_indices) => {
GroupOrderingPartial::try_new(input_schema, order_indices, ordering)
.map(GroupOrdering::Partial)
}
InputOrderMode::Sorted => Ok(GroupOrdering::Full(GroupOrderingFull::new())),
}
}

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a TODO comment to the code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
physical-expr Physical Expressions
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve GroupOrderingPartial performance
5 participants