Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Consider using --to-nodes instead of --to-outputs for kedro runs #2344

Open
Nrbassili opened this issue Dec 5, 2024 · 0 comments
Open

Consider using --to-nodes instead of --to-outputs for kedro runs #2344

Nrbassili opened this issue Dec 5, 2024 · 0 comments
Assignees

Comments

@Nrbassili
Copy link

For LAR and MLAR, we have a node that validates the MLAR and LAR data by checking that the row counts are the same.

With the current cron job and job templates, jobs are generated with the --to-outputs kedro run parameter, which means this validation node is not run for LAR and MLAR datasets.

If we switch to using the --to-nodes kedro run parameter, in the job and cronjob templates, we can set the final validation node to be the target node. This would mean that both MLAR and LAR files are generated in the same kedro run, which will take longer, but allow us to validate the counts at the end. Otherwise, we should consider removing this node, since it isn't used.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants