Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CT-957] Refactor incremental materialization, using new incremental_strategy foundations #512

Open
Tracked by #9290
jtcohen6 opened this issue Jul 29, 2022 · 2 comments
Labels
pkg:dbt-spark Issue affects dbt-spark type:task Eng-driven work

Comments

@jtcohen6
Copy link
Contributor

jtcohen6 commented Jul 29, 2022

Follow up to dbt-labs/dbt-spark#402, which implemented the minimal version to keep the current code working

See dbt-labs/dbt-snowflake#196 for reference implementation:

  • Define valid_incremental_strategies
  • Add to materialization code:
    {#-- Get the incremental_strategy, the macro to use for the strategy, and build the sql --#}
    {% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
    {% set incremental_predicates = config.get('incremental_predicates', none) %}
    {% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}
    {% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': tmp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'predicates': incremental_predicates }) %}
    {% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}
  • Take stock of remaining discrepancies between default + spark incremental materializations, identify opportunities for further refactor + eventual deduplication
@jtcohen6 jtcohen6 added the type:task Eng-driven work label Jul 29, 2022
@github-actions github-actions bot changed the title Refactor incremental materialization, using new incremental_strategy foundations [CT-957] Refactor incremental materialization, using new incremental_strategy foundations Jul 29, 2022
@github-actions
Copy link
Contributor

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.

@github-actions github-actions bot added the Stale Mark an issue or PR as stale, to be closed label Jan 26, 2023
@github-actions github-actions bot closed this as completed Feb 2, 2023
@dbeatty10 dbeatty10 reopened this Jul 11, 2023
@dbeatty10 dbeatty10 removed the Stale Mark an issue or PR as stale, to be closed label Jul 11, 2023
@dbeatty10
Copy link
Contributor

As-of v1.7, dbt-spark currently only supports these incremental strategies:

  • append
  • merge
  • insert_overwrite

So it currently does not support this one:

  • delete+insert

https://docs.getdbt.com/docs/build/incremental-models#supported-incremental-strategies-by-adapter

image

Since dbt-core dbt-adapters has default definitions of append and delete+insert, it should be trivial for dbt-spark to support both of those during implementation of this issue.

@mikealfare mikealfare added the pkg:dbt-spark Issue affects dbt-spark label Jan 13, 2025
@mikealfare mikealfare transferred this issue from dbt-labs/dbt-spark Jan 13, 2025
mikealfare pushed a commit that referenced this issue Jan 14, 2025
* update regex for version bump

* update to prekind

* more renaming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pkg:dbt-spark Issue affects dbt-spark type:task Eng-driven work
Projects
None yet
Development

No branches or pull requests

3 participants