Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Data loaders should be able to inject tags to the graph #611

Open
skrawcz opened this issue Dec 29, 2023 · 0 comments
Open

Data loaders should be able to inject tags to the graph #611

skrawcz opened this issue Dec 29, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Dec 29, 2023

Is your feature request related to a problem? Please describe.
I'm a central platform team. I own some data adapters. I want to be able to inject some tags into the graph on the nodes.

Describe the solution you'd like
Data loaders can be used in many contexts. So I think the only way to get some tags is after instantiation, and a function is called.

Something like

@dataclasses.dataclass
class JSONDataLoader(DataLoader):
    path: str

    @classmethod
    def applicable_types(cls) -> Collection[Type]:
        return [dict]

    def load_data(self, type_: Type) -> Tuple[dict, Dict[str, Any]]:
        with open(self.path, "r") as f:
            return json.load(f), get_file_metadata(self.path)

    @classmethod
    def name(cls) -> str:
        return "json"

    def tags(self) -> dict:  # <------------- add something like this
        return {"some_special_tag": self.path, ...}

Describe alternatives you've considered
It's not critical, but something to think about.

Additional context
If one was to build more capabilities using tag metadata, then we'd want this here.

@skrawcz skrawcz added the enhancement New feature or request label Dec 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant