Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Check that schemas are well-formed upon construction #628

Open
OussamaSaoudi-db opened this issue Jan 9, 2025 · 0 comments
Open
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@OussamaSaoudi-db
Copy link
Collaborator

OussamaSaoudi-db commented Jan 9, 2025

Please describe why this is necessary.

Delta disallows schemas that have two fields that differ only in case. For example, a schema with the fields id and ID is not allowed. This is also checked in the delta-spark implementation here.

Describe the functionality you are proposing.

Come up with a way to construct StructTypes that checks they are well-formed as defined by the reference implementation.

Additional context

For context on how this issue originally came up, look here.

Here's a non-exhaustive list of cases that should perform this check

  • When a Schema is parsed from a string. This happens after parsing Metadata actions for instance.
  • When a schema is passed to the kernel using the EngineSchemaVisitor in ffi/src/schema.rs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

2 participants