Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[590] Add Iceberg HMS Catalog Sync implementation #600

Open
wants to merge 7 commits into
base: 590-CatalogSync
Choose a base branch
from

Conversation

kroushan-nit
Copy link
Contributor

@kroushan-nit kroushan-nit commented Dec 13, 2024

Important Read

  • Please ensure the GitHub issue is mentioned at the beginning of the PR

What is the purpose of the pull request

Add HMS catalog sync client implementation for Iceberg

Brief change log

Verify this pull request

  • Unit tests

@kroushan-nit kroushan-nit marked this pull request as draft December 13, 2024 18:53
@kroushan-nit kroushan-nit marked this pull request as ready for review December 17, 2024 10:32
@vinishjail97 vinishjail97 force-pushed the 590-CatalogSync branch 3 times, most recently from a723014 to ced6288 Compare December 19, 2024 22:12
@vinishjail97 vinishjail97 changed the title Add Iceberg HMS Catalog Sync implementation [590] Add Iceberg HMS Catalog Sync implementation Dec 20, 2024
@vinishjail97 vinishjail97 force-pushed the 590-CatalogSync branch 2 times, most recently from 893b1a6 to eefbdd2 Compare December 26, 2024 19:37
@kroushan-nit kroushan-nit force-pushed the 590-IcebergHMSSync branch 2 times, most recently from 970c2ef to 88c0471 Compare December 27, 2024 08:00
pom.xml Outdated Show resolved Hide resolved
<hudi.version>0.14.0</hudi.version>
<aws.version>2.28.22</aws.version>
<hive.version>2.3.9</hive.version>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a 4.X.X release, can we try using that or at least the 3.X.X line? I am worried that any vulnerability patches will not make it to the N-2 major version line.

@kroushan-nit kroushan-nit force-pushed the 590-IcebergHMSSync branch 3 times, most recently from 34f18a0 to 25bf942 Compare December 31, 2024 11:50

<!-- Iceberg dependencies -->
<dependency>
<groupId>org.apache.iceberg</groupId>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this only used to get a class name as a string?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is also used during HiveMetaHook#preCreateTable as part of creating table

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you link me to where that is happening in this branch?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is not part of this branch, rather with storage_handler property set, metastore client executes the preCreateTable method of the HiveMetaHook associated with the HiveIcebergStorageHandler during createTable operation

image

HMS client createTable: https://github.com/apache/hive/blob/rel/release-2.3.9/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L741

HiveMetahookLoader: https://github.com/apache/hive/blob/rel/release-2.3.9/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java#L3579

}
}

public static Table newHmsTable(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should there be a unit test for this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test should be in a test class for HMSCatalogTableBuilderFactory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants