Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor: Move all Protocol and Metadata validation to TableConfiguration #650

Open
7 tasks
OussamaSaoudi opened this issue Jan 17, 2025 · 0 comments
Open
7 tasks

Comments

@OussamaSaoudi
Copy link
Collaborator

OussamaSaoudi commented Jan 17, 2025

Motivation

TableConfiguration should be the single point in the kernel where we validate a table's protocol and metadata. This makes it easy to ensure these two actions are mutually compatible. This also allows kernel to draw a clear distinction between raw data (Protocol & Metadata), and validated data in TableConfiguration.

Tasks

  • Protocol's try_new method performs some validation on the set of reader features and checks that they are supported. Move this validation to TableConfiguration::try_new.
  • Ensure that the set of reader features is a superset of writer features.
  • Eagerly parse the reader and writer features so that we do not parse them repeatedly such as in ensure_read_supported.
  • Add validation for Metadata in the TableConfiguration constructor. Ex: it should check partition columns are present in the schema.
  • Move any tests that check protocol and metadata validity to instead check the TableConfiguration. Ex: the tests for protocol such as test_ensure_read_supported
  • Create a framework for testing valid/invalid protocol/metadata configurations.
  • Migrate integration tests in table_changes/mod.rs that check failed configurations to instead be unit tests for TableConfiguration. This ensures that we cover all edge cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant