Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[IcebergIO] Use InternalRecordWrapper partition util #33701

Merged
merged 3 commits into from
Jan 23, 2025

Conversation

ahmedabu98
Copy link
Contributor

Previously we were manually doing conversions to determine a record's partition. We recently found that Iceberg has existing public utils that do this; this change switches to use those utils.

Fixes #32865

@ahmedabu98
Copy link
Contributor Author

assign set of reviewers

Copy link
Contributor

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @m-trieu for label java.
R: @Abacn for label build.
R: @Abacn for label io.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@svetakvsundhar svetakvsundhar self-assigned this Jan 21, 2025
@svetakvsundhar svetakvsundhar self-requested a review January 21, 2025 21:23
@svetakvsundhar svetakvsundhar removed their assignment Jan 21, 2025
@@ -465,6 +486,10 @@ public void testIdentityPartitioning() throws IOException {
.identity("float")
.identity("double")
.identity("str")
.identity("date")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for my knowledge, it's unclear to me how testIdentityPartitioning() was passing when identity should support Any source type. Is it worth adding an integration test to ensure just the Write works when partitioning by just date, as well as Any?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test used to not include date and the other time types. This PR adds adds them to the test (your comment is actually positioned on the lines that add them).

@ahmedabu98 ahmedabu98 merged commit fa0152b into apache:master Jan 23, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug]: IcebergIO cannot write data into an hourly partitioned table
2 participants