Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modify fct_model_directories model to handle staging model dependent on multiple sources #525

Draft
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

BradCr
Copy link
Contributor

@BradCr BradCr commented Jan 19, 2025

This is a:

  • bug fix PR with no breaking changes
  • new functionality

Link to Issue

Closes #111

Description & motivation

Description from the Issue: In fct_model_directories we're specifying staging_models that have an incorrect file path based on their upstream source dependencies. However, if a staging model is dependent on multiple sources (against our best practices, and will cause a warning for other checks in this package, but alas it could still happen) then there will be multiple records for the staging model potentially indicating multiple different values for change_file_path_to. This means that you may be unable to resolve the warnings from this check in the case where your staging model depends on 2 different sources (defined in 2 different directories).

I added a CTE to count number of sources in a staging model, and then listagg() them into a single change_file_path_to column explaining the user should split into two staging models in their appropriate folders..

I haven't added any scenarios to the integration tests yet, which is why I have put this PR into draft mode for now.

I haven't yet updated the documentation for fct_model_directories, but that is something I can do before requesting final review of the PR.

Integration Test Screenshot

Not available yet.

Checklist

  • I have verified that these changes work locally on the following warehouses (Note: it's okay if you do not have access to all warehouses, this helps us understand what has been covered)
    • BigQuery
    • Postgres
    • Redshift
    • Snowflake
    • Databricks
    • DuckDB
    • Trino/Starburst
  • I have updated the README.md (if applicable)
  • I have added tests & descriptions to my models (and macros if applicable)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

how to handle staging models that depend on multiple sources in fct_staging_directories
1 participant