[Kernel] Load the protocol and metadata from the CRC files when available #4077

huan233usc · 2025-01-21T18:28:30Z

Which Delta project/connector is this regarding?

Description

CP crc loading code to master branch from "kernel-20250115-crc-optimization"
PR created using git cherry-pick 7d32f66d9efd1dca41ed2a45cf259525cbdd8952

How was this patch tested?

Does this PR introduce any user-facing changes?

No

… in DeltaLog Initial read current version CRC w test to verify wip end-2-end working + tests Co-authored-by: Allison Portis <[email protected]> Co-authored-by: Venki Korukanti <[email protected]>

scottsand-db

Looks great! Left some comments.

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/VersionStats.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/ChecksumReader.java

scottsand-db · 2025-01-21T19:18:44Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/LogReplay.java

+        ChecksumReader.getVersionStats(
+            engine, logSegment.logPath, snapshotVersion, crcSearchLowerBound);
+    if (versionStatsOpt.isPresent()) {
+      // We found the protocol and metadata for the version we are looking for


can we add a check that versionStateOpt.get.version is > (or >=, whatever is corect) than the snapshot hint?

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/LogReplay.java

...el/kernel-defaults/src/test/scala/io/delta/kernel/defaults/LogReplayEngineMetricsSuite.scala

kernel/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaTableReadsSuite.scala

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/LogReplay.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/ChecksumReader.java

allisonport-db · 2025-01-22T19:01:07Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/ChecksumReader.java

+    try (CloseableIterator<FileStatus> crcFiles =
+        engine.getFileSystemClient().listFrom(lowerBoundFilePath.toString())) {
+      List<FileStatus> crcFilesList = new ArrayList<>();
+      crcFiles
+          .filter(file -> isChecksumFile(new Path(file.getPath())))
+          .forEachRemaining(crcFilesList::add);


@scottsand-db I wonder if this can be refactored to use any of the log listing methods in your PR?

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/LogReplay.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/FileNames.java

kernel/kernel-defaults/src/test/scala/io/delta/kernel/defaults/DeltaTableReadsSuite.scala

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java

scottsand-db

LGTM!

allisonport-db · 2025-01-24T22:20:02Z

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/InternalUtils.java

+  public static <T> List<T> toFilteredList(
+      CloseableIterator<T> iterator, Function<T, Boolean> filter) {
+    List<T> result = new ArrayList<>();
+    iterator.filter(filter).forEachRemaining(result::add);
+    return result;
+  }


Sorry nit, I think it's easier if this is just toList and we can do the iterator filtering in the outer call. This way we can re-use this even when we don't want to filter

i.e. toList(iterator.filter(filter))

allisonport-db

Just a few minor remaining comments then LGTM

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/ChecksumReader.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/LogReplay.java

scottsand-db

Looks great! Unfortunately, takeWhile definitely needs to be its own PR.

kernel/kernel-api/src/main/java/io/delta/kernel/utils/CloseableIterator.java

kernel/kernel-api/src/test/scala/io/delta/kernel/utils/CloseableIteratorSuite.scala

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/InternalUtils.java

[Kernel] Support for loading protocol and metadata from checksum file…

dd9fe7b

… in DeltaLog Initial read current version CRC w test to verify wip end-2-end working + tests Co-authored-by: Allison Portis <[email protected]> Co-authored-by: Venki Korukanti <[email protected]>

huan233usc mentioned this pull request Jan 21, 2025

[Kernel] Load the protocol and metadata from the CRC files when available #4071

Closed

5 tasks

scottsand-db requested review from scottsand-db, vkorukanti and allisonport-db and removed request for vkorukanti January 21, 2025 19:05

scottsand-db assigned huan233usc Jan 21, 2025

scottsand-db requested changes Jan 21, 2025

View reviewed changes

huan233usc added 4 commits January 21, 2025 15:31

refactor according to the comments

df12241

add logging and fix comments

a404846

move comments

1e410c5

update comments

1994b2e

huan233usc force-pushed the cp-crc branch from 6c44a28 to 1994b2e Compare January 22, 2025 01:12

huan233usc requested a review from scottsand-db January 22, 2025 17:46

update comments

ddc0815

scottsand-db requested changes Jan 22, 2025

View reviewed changes

replace static test table with test generated

02067f3

allisonport-db reviewed Jan 22, 2025

View reviewed changes

huan233usc added 7 commits January 22, 2025 12:05

use optional in building crc and use rfii in crc test

dec4938

resolve comments

6eff126

update comments

69c29df

fix test

fec680e

updated param name

60981a6

update doc

97bc44f

fix idention

2f5b1a2

scottsand-db reviewed Jan 22, 2025

View reviewed changes

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java Outdated Show resolved Hide resolved

scottsand-db reviewed Jan 22, 2025

View reviewed changes

kernel/kernel-api/src/main/java/io/delta/kernel/internal/replay/CRCInfo.java Outdated Show resolved Hide resolved

huan233usc added 2 commits January 22, 2025 13:25

update the doc to reflect nullness

c7bf343

fix javafmt

dbf4490

huan233usc requested a review from scottsand-db January 22, 2025 22:18

format java

588b464

scottsand-db approved these changes Jan 24, 2025

View reviewed changes

allisonport-db reviewed Jan 24, 2025

View reviewed changes

update internal utils to move filter away

155cf2b

huan233usc requested a review from allisonport-db January 24, 2025 22:27

allisonport-db requested changes Jan 28, 2025

View reviewed changes

fix comment

8fe533b

scottsand-db self-requested a review January 28, 2025 21:51

huan233usc added 2 commits January 28, 2025 13:51

take while

119d0e0

adding header

c189537

huan233usc requested a review from allisonport-db January 28, 2025 21:55

huan233usc added 2 commits January 28, 2025 13:56

fix typo

cf54166

fix typo

21ed78f

scottsand-db requested changes Jan 28, 2025

View reviewed changes

huan233usc added 7 commits January 29, 2025 15:23

switch to checkpoint opt

5bf479e

remove unused import

8a96791

merge with conflict

9882740

resolving conflict 1 - remove unnecessary files

d05450d

resolving conflict 2 fix tests

4e7078d

resolving conflict 3 resolve accidental deleted suites

2408be8

empty line

aa82629

scottsand-db self-requested a review January 30, 2025 21:11

huan233usc added 2 commits January 30, 2025 13:18

fix year

5b6c03b

fix year

bdba22a

scottsand-db approved these changes Jan 30, 2025

View reviewed changes

allisonport-db requested changes Jan 30, 2025

View reviewed changes

kernel/kernel-api/src/main/java/io/delta/kernel/internal/util/InternalUtils.java Outdated Show resolved Hide resolved

use inmemory list

a1611b8

allisonport-db approved these changes Jan 30, 2025

View reviewed changes

scottsand-db merged commit 92a8a22 into delta-io:master Jan 31, 2025
19 checks passed

huan233usc mentioned this pull request Feb 1, 2025

[Kernel]Remove two file IO in CRC loading by reusing log listing #4112

Draft

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Kernel] Load the protocol and metadata from the CRC files when available #4077

[Kernel] Load the protocol and metadata from the CRC files when available #4077

huan233usc commented Jan 21, 2025

scottsand-db left a comment

scottsand-db Jan 21, 2025

allisonport-db Jan 22, 2025

scottsand-db left a comment

allisonport-db Jan 24, 2025

allisonport-db left a comment

scottsand-db left a comment

[Kernel] Load the protocol and metadata from the CRC files when available #4077

[Kernel] Load the protocol and metadata from the CRC files when available #4077

Conversation

huan233usc commented Jan 21, 2025

Which Delta project/connector is this regarding?

Description

How was this patch tested?

Does this PR introduce any user-facing changes?

scottsand-db left a comment

Choose a reason for hiding this comment

scottsand-db Jan 21, 2025

Choose a reason for hiding this comment

allisonport-db Jan 22, 2025

Choose a reason for hiding this comment

scottsand-db left a comment

Choose a reason for hiding this comment

allisonport-db Jan 24, 2025

Choose a reason for hiding this comment

allisonport-db left a comment

Choose a reason for hiding this comment

scottsand-db left a comment

Choose a reason for hiding this comment