Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spark][TEST-ONLY] Test identity high water mark inserts when target table has no high water mark #4049

Merged
merged 2 commits into from
Jan 16, 2025

Conversation

c27kwan
Copy link
Contributor

@c27kwan c27kwan commented Jan 14, 2025

Which Delta project/connector is this regarding?

  • Spark
  • Standalone
  • Flink
  • Kernel
  • Other (fill in here)

Description

Test only PR. Adds some basic additional tests for inserting data into tables when the Identity Column high water mark isn't already defined. Split from the PR making identity column high water mark updates more consistent: #3989

How was this patch tested?

New tests pass.

Does this PR introduce any user-facing changes?

No.

Copy link
Contributor

@zhipengmao-db zhipengmao-db left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for adding test coverage!

@c27kwan c27kwan changed the title [TEST-ONLY] Test identity high water mark inserts when target table h… [Spark][TEST-ONLY] Test identity high water mark inserts when target table h… Jan 15, 2025
@c27kwan c27kwan changed the title [Spark][TEST-ONLY] Test identity high water mark inserts when target table h… [Spark][TEST-ONLY] Test identity high water mark inserts when target table has no high water mark Jan 15, 2025
@c27kwan
Copy link
Contributor Author

c27kwan commented Jan 15, 2025

I don't know how to re-trigger the jobs, but the failures are completely orthogonal due to this being a test only PR. I've run this locally and it passes.

@vkorukanti vkorukanti merged commit 5d794e5 into delta-io:master Jan 16, 2025
16 of 19 checks passed
huan233usc pushed a commit to huan233usc/delta that referenced this pull request Jan 17, 2025
…table has no high water mark (delta-io#4049)

#### Which Delta project/connector is this regarding?

- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)

## Description
Test only PR. Adds some basic additional tests for inserting data into
tables when the Identity Column high water mark isn't already defined.
Split from the PR making identity column high water mark updates more
consistent: delta-io#3989

## How was this patch tested?

New tests pass.

## Does this PR introduce _any_ user-facing changes?
No.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants