generate recon config from table definition #1433
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
sriram251-code
added
the
feat/recon
| with open(file_name, 'w') as json_file: | ||
| json_file.write(json_data) | ||
| except (TypeError, IOError) as e: | ||
| print(f"Failed to generate JSON file: {e}") |
| from databricks.labs.remorph.config import TableRecon | ||
|
|
||
|
|
||
| def generate_json_file(table_recon: TableRecon, file_name: str): |
There was a problem hiding this comment.
TableRecon is the target You will need to introduce TableDefinition.
from dataclasses import dataclass, field
from typing import List, Optional
@dataclass
class TableDefinition:
catalog: str
schema: str
table: str
location: Optional[str] = None
tableFormat: Optional[str] = None
viewText: Optional[str] = None
columns: List[StructField] = field(default_factory=list)
sizeGb: int = 0
comment: Optional[str] = None
There was a problem hiding this comment.
I would suggest to add primary_keys: list[str] | None = None to it too.
There was a problem hiding this comment.
We may not use StructField for field information in TableDefinition. Because the dataType field in it is of type DataType ; which is tied to Spark. When we bring metadata from 3rd party systems, they may not have a direct equivalent for Spark data types. So we may want to use our own dataclass to store column information which is loosely structured around StructField but uses string for dataType. If needed, we can have our dataclass -> StructField transformation function.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This pull request introduces functionality to generate a JSON file from a
TableReconobject and includes a unit test for this new feature.