[docs] Add documentation for file-based metastore

[steveburnett]

Description

Add how to configure Presto to use a file-based hms to installation/deployment.rst.

Motivation and Context

@ethanyzhang suggested this would be a good addition to the Presto documentation in an internal discussion that @nmahadevuni contributed the configuration in. I discussed where such information would fit best in the Presto documentation with @tdcmeehan.

Impact

Documentation. Readers wanting to try out Presto quickly can bypass the need for the steps in Configure Hive MetaStore.

Test Plan

Local doc build. Screenshot with existing text above and below included for context.

Contributor checklist

• Please make sure your submission complies with our contributing guide, in particular code style and commit standards.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

Please follow release notes guidelines and fill in the release notes below.

== RELEASE NOTES ==

General Changes
* Add documentation for file-based Hive metastore to :doc:`/connector/file-based-metastore`.

[majetideepak]

@steveburnett I have an issue here with some more details
#19112
We need to list the restrictions as well. file-based metastore does not support partitioning for example. @nmahadevuni should confirm

[imjalpreet]

The metastore concept applies only to Hive and Lakehouse Connectors (Iceberg, Delta, and Hudi), so we should probably mention it.

Additionally, these properties are catalog properties and must be added to etc/catalog/catalog_name.properties valid only for the above connectors.

[steveburnett]

Thanks, will do!

[imjalpreet]

file-based meta store does not support partitioning

@majetideepak I haven't used the file-based Hive metastore much myself, but did you encounter any issues when trying to create partitioned tables?

Ideally, it should be possible and I can see that partition-specific metadata calls are implemented even for the FileHiveMetastore.

https://github.com/prestodb/presto/blob/master/presto-hive-metastore/src/main/java/com/facebook/presto/hive/metastore/file/FileHiveMetastore.java

[ethanyzhang]

@imjalpreet should this be in the hive.properties or config.properties?

[imjalpreet]

should this be in the hive.properties or config.properties?

@ethanyzhang These should be in the catalog properties file. I also added a review comment above.

[steveburnett]

@steveburnett I have an issue here with some more details #19112 We need to list the restrictions as well. file-based metastore does not support partitioning for example. @nmahadevuni should confirm

Thanks for the additional information @majetideepak!

With the new information in #19112, I am going to move this topic from where I initially put it in this PR as a small topic in Deploying Presto.

I thought about moving it into the Hive Connector doc, but as it is relevant to "Hive and Lakehouse Connectors (Iceberg, Delta, and Hudi)" I think I will move it to its own page in /installation and include Deepak's instructions how to use it, which will be a big help to readers.

[imjalpreet]

Would it make more sense to include this in the connectors section?

Right now, metastore properties are spread across different parts of the Hive Connector documentation, mainly in Hive Configuration Properties, Metastore Configuration Properties, and Glue Configuration Properties.

Perhaps we could extract the metastore-related documentation into a separate subsection, similar to Hive Security. This would make sense since metastore properties are relevant not just to Hive, but also to other connectors like Iceberg, Delta, and Hudi.

[majetideepak]

Ideally, it should be possible and I can see that partition-specific metadata calls are implemented even for the FileHiveMetastore.

@imjalpreet I remember seeing some issue with partitions and file metastore. It could be due to my setup. I think its good to test this once before documenting.

[hantangwangd]

I use file bases HMS for development environment, just confirmed again that it supports Hive partitioned tables. Or maybe you encountered some specific problems when using partitioned tables with file bases HMS @majetideepak?

[hantangwangd]

Should we also mentioned that it is also very suitable for developing purpose?

[majetideepak]

@hantangwangd If you use it, then it's good. My issue was likely related to my setup.

[nmahadevuni]

The path need not be relative to the installation directory. It can be any path on the local filesystem.

[nmahadevuni]

@majetideepak I have used partitioned tables too recently with file based HMS. I didn't have any issues.

[steveburnett]

Hi everyone, thanks for your feedback! I took @imjalpreet's suggestion and moved this to a separate page in /connector, corrected the directory path that @nmahadevuni noted, added text describing the supported connectors and corrected the instructions for the properties file to the connector property file, and added usage examples from @majetideepak's #19112.

I welcome everyone to review again and comment with new corrections and additions.

[nmahadevuni]

It is currently supported only on local file system. S3 is not supported. Please remove this.

[steveburnett]

I sourced that from #19112, but if it's not supported then I can remove it, yes.

[majetideepak]

I verified everything in #19112. Does not hurt to test again since its been a while.

[hantangwangd]

Currently it do not support HDFS or S3, but we can make some small changes to let it support all of them for test or development environment (that is to say, not safe in production environment).

[steveburnett]

Okay, that helps!

If HDFS and S3 are not currently supported, then I will remove the reference to S3 and write the doc as "supported only on local file system". When the "changes [are made] to let it support all of them for test or development environment", I can update the doc then to add mention that HDFS and S3 are optional for test or dev only.

[nmahadevuni]

Let's specify that this happens if the user gave /data/hive_data/ as the path for hive.metastore.catalog.dir

[nmahadevuni]

Did we verify this with a test?

[steveburnett]

I took this content from #19112, didn't test it personally. @majetideepak, does this work as described?

[majetideepak]

I tested this. I also now remember that this is where partitioning gets tricky.

[majetideepak]

@steveburnett, @hantangwangd, @nmahadevuni
The file-based metastore approach is also useful for reading existing files.
I tested all the content in #19112
It does not cover partitioning. We could cover partitioning in a separate PR.

[nmahadevuni]

Since we are mentioning one can provide .prestoSchema and .prestoPermissions for existing data, is there any format or steps on how to create these? @majetideepak

[steveburnett]

Hi everyone! I revised based on the feedback from @nmahadevuni, @majetideepak, and @hantangwangd. PTAL when you can.

Specific open questions I haven't addressed in this update:

• what I should say or remove from the current draft about partitioning. Should I add a sentence in the Overview "Partitioning with file-based metastores is not supported."?

• what I should add about "provide .prestoSchema and .prestoPermissions files", or remove that line entirely?

[majetideepak]

@steveburnett Let's remove Reading Existing Data Files with a File-based Metastore from this PR since both the open questions are related to that.
I believe @imjalpreet mentioned that partitioning works when you create the table with file based metastore.

[steveburnett]

@steveburnett Let's remove Reading Existing Data Files with a File-based Metastore from this PR since both the open questions are related to that. I believe @imjalpreet mentioned that partitioning works when you create the table with file based metastore.

Done and done, thanks! PTAL.

[hantangwangd]

Thanks for the work @steveburnett.

[nmahadevuni]

Thank you @steveburnett.

[imjalpreet]

@steveburnett should we create a dedicated sub-section for Hive Metastore instead of just “File Hive Metastore”? We could consolidate all metastore-related information from the Hive connector documentation into this section to keep it organized and less scattered.

[imjalpreet]

nit: maybe this can be removed since it's expected behaviour, we could have mentioned if it wasn't supported.