From d8cf447edd8fc5d46507dacbd4293c6974b31a23 Mon Sep 17 00:00:00 2001 From: Josh Andrews <65506623+codigo-ergo-sum@users.noreply.github.com> Date: Mon, 8 Nov 2021 20:55:37 -0800 Subject: [PATCH 1/5] =?UTF-8?q?Fixing=20an=20issue=20where=20the=20mutuall?= =?UTF-8?q?y=20exclusive=20range=20test=20doesn't=20work=20=E2=80=A6=20(#4?= =?UTF-8?q?30)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Fixing an issue where the mutually exclusive range test doesn't work with store failures with a multi-column concatenated partition by clause. As per: https://github.com/dbt-labs/dbt-utils/issues/423 * Bumping dbt version to take into account CircleCI already being upgrade to dbt 1.0b as per Joel Labes * Further tweak to version number checks to try to fix CircleCI * Update pull_request_template.md (#438) * Update pull_request_template.md * Update .github/pull_request_template.md * Add 0.8.0 changelog block * Add changelog for mutually excl ranges change --- .github/pull_request_template.md | 5 ++++- CHANGELOG.md | 7 +++++++ macros/schema_tests/mutually_exclusive_ranges.sql | 2 +- 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 32ec56ed..7d8278dd 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -14,7 +14,10 @@ Describe your changes, and why you're making them. - [ ] Postgres - [ ] Redshift - [ ] Snowflake -- [ ] I have "dispatched" any new macro(s) so non-core adapters can also use them (e.g. [the `star()` source](https://github.com/fishtown-analytics/dbt-utils/blob/master/macros/sql/star.sql)) +- [ ] I followed guidelines to ensure that my changes will work on "non-core" adapters by: + - [ ] dispatching any new macro(s) so non-core adapters can also use them (e.g. [the `star()` source](https://github.com/fishtown-analytics/dbt-utils/blob/master/macros/sql/star.sql)) + - [ ] using the `limit_zero()` macro in place of the literal string: `limit 0` + - [ ] using `dbt_utils.type_*` macros instead of explicit datatypes (e.g. `dbt_utils.type_timestamp()` instead of `TIMESTAMP` - [ ] I have updated the README.md (if applicable) - [ ] I have added tests & descriptions to my models (and macros if applicable) - [ ] I have added an entry to CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 150b71a2..85d34510 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,10 @@ +# dbt-utils v0.8.0 +## 🚨 Breaking changes +- The partition column in the `mutually_exclusive_ranges` test is now always called `partition_by_col`. This enables compatibility with `--store-failures` when multiple columns are concatenated together. If you have models built on top of the failures table, update them to reflect the new column name. ([#423](https://github.com/dbt-labs/dbt-utils/issues/423), [#430](https://github.com/dbt-labs/dbt-utils/pull/430)) + +## Contributors: +- [codigo-ergo-sum](https://github.com/codigo-ergo-sum) (#430) + # dbt-utils v0.7.4b1 This is a compatibility release in preparation for `dbt-core` v1.0.0 (🎉). When dbt-core 1.0.0 hits release candidate status, we will release the final version of 0.7.4 diff --git a/macros/schema_tests/mutually_exclusive_ranges.sql b/macros/schema_tests/mutually_exclusive_ranges.sql index f2da3b95..f88efbe4 100644 --- a/macros/schema_tests/mutually_exclusive_ranges.sql +++ b/macros/schema_tests/mutually_exclusive_ranges.sql @@ -35,7 +35,7 @@ with window_functions as ( select {% if partition_by %} - {{ partition_by }}, + {{ partition_by }} as partition_by_col, {% endif %} {{ lower_bound_column }} as lower_bound, {{ upper_bound_column }} as upper_bound, From 296de1f564fc61b3da6994fb4718833bc00f24aa Mon Sep 17 00:00:00 2001 From: Adam Stone <31513760+foundinblank@users.noreply.github.com> Date: Tue, 9 Nov 2021 21:43:33 +0000 Subject: [PATCH 2/5] Add handling for `android-app://` in `get_url_host()` macro (#426) * Add handling for `android-app://` * change to regexp and update integration test * forgot comma * try this * revert * try this way * copy/pasting is not always smart * try | safe * let's try this * go back to original * fix csv * Add line breaks back in * one more line break * Update changelog --- CHANGELOG.md | 2 ++ integration_tests/data/web/data_url_host.csv | 5 ++++- macros/web/get_url_host.sql | 13 +++++++------ 3 files changed, 13 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 85d34510..38cbebb9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,10 +16,12 @@ This is a compatibility release in preparation for `dbt-core` v1.0.0 (🎉). Whe - Bump `require-dbt-version` to have an upper bound of `'<=1.0.0'`. - Url link fixes within the README for `not_constant`, `dateadd`, `datediff` and updated the header `Logger` to `Jinja Helpers`. ([#431](https://github.com/dbt-labs/dbt-utils/pull/431)) - Fully qualified a `cte_name.*` in the `equality` test to avoid an Exasol error ([#420](https://github.com/dbt-labs/dbt-utils/pull/420)) +- `get_url_host()` macro now correctly handles URLs beginning with `android-app://` ([#426](https://github.com/dbt-labs/dbt-utils/pull/426)) ## Contributors: - [joemarkiewicz](https://github.com/fivetran-joemarkiewicz) - [TimoKruth](https://github.com/TimoKruth) +- [foundinblank](https://github.com/foundinblank) # dbt-utils v0.7.3 diff --git a/integration_tests/data/web/data_url_host.csv b/integration_tests/data/web/data_url_host.csv index c32fbf8d..36daaa00 100644 --- a/integration_tests/data/web/data_url_host.csv +++ b/integration_tests/data/web/data_url_host.csv @@ -1,4 +1,7 @@ original_url,parsed_url www.google.co.uk?utm_source=google&utm_medium=cpc&utm_campaign=spring-summer,www.google.co.uk http://witanddelight.com/2018/01/tips-tricks-how-run-half-marathon-first-time/,witanddelight.com -https://www.nytimes.com/2018/01/01/blog,www.nytimes.com \ No newline at end of file +https://www.nytimes.com/2018/01/01/blog,www.nytimes.com +android-app://m.facebook.com/,m.facebook.com +docs.nytimes.com/2021/01/01/index.js?utm_source=google,docs.nytimes.com +https://m.facebook.com/,m.facebook.com diff --git a/macros/web/get_url_host.sql b/macros/web/get_url_host.sql index b1e8e4de..43a58cc4 100644 --- a/macros/web/get_url_host.sql +++ b/macros/web/get_url_host.sql @@ -4,23 +4,24 @@ {% macro default__get_url_host(field) -%} -{%- set parsed = +{%- set parsed = dbt_utils.split_part( dbt_utils.split_part( dbt_utils.replace( - dbt_utils.replace(field, "'http://'", "''" + dbt_utils.replace( + dbt_utils.replace(field, "'android-app://'", "''" + ), "'http://'", "''" ), "'https://'", "''" ), "'/'", 1 ), "'?'", 1 ) - + -%} - + {{ dbt_utils.safe_cast( parsed, dbt_utils.type_string() )}} - -{%- endmacro %} \ No newline at end of file +{%- endmacro %} From 2ed471bb3a1f9f6a748c4235ead5b71d7b910f95 Mon Sep 17 00:00:00 2001 From: jmriego Date: Thu, 11 Nov 2021 03:12:50 +0100 Subject: [PATCH 3/5] return default in get_column_values if no execute (#386) * return default in get_column_values if no execute * Update CHANGELOG.md * Update macros/sql/get_column_values.sql Co-authored-by: Jason Ganz * Update require-dbt-version Co-authored-by: Joel Labes Co-authored-by: Jason Ganz --- CHANGELOG.md | 1 + macros/sql/get_column_values.sql | 6 ++++-- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 38cbebb9..6c1d3d6a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -102,6 +102,7 @@ If you were relying on the position to match up your optional arguments, this ma ## Under the hood * Update the default implementation of concat macro to use `||` operator ([#373](https://github.com/fishtown-analytics/dbt-utils/pull/314) from [@ChristopheDuong](https://github.com/ChristopheDuong)). Note this may be a breaking change for adapters that support `concat()` but not `||`, such as Apache Spark. - Use `power()` instead of `pow()` in `generate_series()` and `haversine_distance()` as they are synonyms in most SQL dialects, but some dialects only have `power()` ([#354](https://github.com/fishtown-analytics/dbt-utils/pull/354) from [@swanderz](https://github.com/swanderz)) +- Make `get_column_values` return the default value passed as a parameter instead of an empty string before compilation ([#304](https://github.com/dbt-labs/dbt-utils/pull/386) from [@jmriego](https://github.com/jmriego) # dbt-utils v0.6.6 diff --git a/macros/sql/get_column_values.sql b/macros/sql/get_column_values.sql index 1f79a455..e5249374 100644 --- a/macros/sql/get_column_values.sql +++ b/macros/sql/get_column_values.sql @@ -3,10 +3,12 @@ {% endmacro %} {% macro default__get_column_values(table, column, order_by='count(*) desc', max_records=none, default=none) -%} - +{% if default is none %} + {% set default = [] %} +{% endif %} {#-- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. #} {%- if not execute -%} - {{ return('') }} + {{ return(default) }} {% endif %} {%- set target_relation = adapter.get_relation(database=table.database, From cffe8d5313dd7d4e2ad1fac25a2257a3ddce41e1 Mon Sep 17 00:00:00 2001 From: Jamie Rodriguez <65564846+fivetran-jamie@users.noreply.github.com> Date: Thu, 25 Nov 2021 17:26:02 -0800 Subject: [PATCH 4/5] add prefix and suffix arguments to star macro (#436) * add prefix and suffix to star macro * add tests * snowflake casing? --- README.md | 2 +- .../data/sql/data_star_prefix_suffix_expected.csv | 4 ++++ integration_tests/models/sql/schema.yml | 5 +++++ .../models/sql/test_star_prefix_suffix.sql | 13 +++++++++++++ macros/sql/star.sql | 8 ++++---- 5 files changed, 27 insertions(+), 5 deletions(-) create mode 100644 integration_tests/data/sql/data_star_prefix_suffix_expected.csv create mode 100644 integration_tests/models/sql/test_star_prefix_suffix.sql diff --git a/README.md b/README.md index 72861e16..ca46d193 100644 --- a/README.md +++ b/README.md @@ -726,7 +726,7 @@ group by 1,2,3 ``` #### star ([source](macros/sql/star.sql)) -This macro generates a list of all fields that exist in the `from` relation, excluding any fields listed in the `except` argument. The construction is identical to `select * from {{ref('my_model')}}`, replacing star (`*`) with the star macro. This macro also has an optional `relation_alias` argument that will prefix all generated fields with an alias. +This macro generates a list of all fields that exist in the `from` relation, excluding any fields listed in the `except` argument. The construction is identical to `select * from {{ref('my_model')}}`, replacing star (`*`) with the star macro. This macro also has an optional `relation_alias` argument that will prefix all generated fields with an alias (`relation_alias`.`field_name`). The macro also has optional `prefix` and `suffix` arguments, which will be appropriately concatenated to each field name in the output (`prefix` ~ `field_name` ~ `suffix`). **Usage:** ```sql diff --git a/integration_tests/data/sql/data_star_prefix_suffix_expected.csv b/integration_tests/data/sql/data_star_prefix_suffix_expected.csv new file mode 100644 index 00000000..fd34199b --- /dev/null +++ b/integration_tests/data/sql/data_star_prefix_suffix_expected.csv @@ -0,0 +1,4 @@ +prefix_field_1_suffix,prefix_field_2_suffix,prefix_field_3_suffix +a,b,c +d,e,f +g,h,i diff --git a/integration_tests/models/sql/schema.yml b/integration_tests/models/sql/schema.yml index 0e8b2497..5b341457 100644 --- a/integration_tests/models/sql/schema.yml +++ b/integration_tests/models/sql/schema.yml @@ -106,6 +106,11 @@ models: - dbt_utils.equality: compare_model: ref('data_star_expected') + - name: test_star_prefix_suffix + tests: + - dbt_utils.equality: + compare_model: ref('data_star_prefix_suffix_expected') + - name: test_surrogate_key tests: - assert_equal: diff --git a/integration_tests/models/sql/test_star_prefix_suffix.sql b/integration_tests/models/sql/test_star_prefix_suffix.sql new file mode 100644 index 00000000..ae1c64fe --- /dev/null +++ b/integration_tests/models/sql/test_star_prefix_suffix.sql @@ -0,0 +1,13 @@ +{% set prefix_with = 'prefix_' if target.type != 'snowflake' else 'PREFIX_' %} +{% set suffix_with = '_suffix' if target.type != 'snowflake' else '_SUFFIX' %} + +with data as ( + + select + {{ dbt_utils.star(from=ref('data_star'), prefix=prefix_with, suffix=suffix_with) }} + + from {{ ref('data_star') }} + +) + +select * from data \ No newline at end of file diff --git a/macros/sql/star.sql b/macros/sql/star.sql index d659e7cb..ffe48946 100644 --- a/macros/sql/star.sql +++ b/macros/sql/star.sql @@ -1,8 +1,8 @@ -{% macro star(from, relation_alias=False, except=[]) -%} - {{ return(adapter.dispatch('star', 'dbt_utils')(from, relation_alias, except)) }} +{% macro star(from, relation_alias=False, except=[], prefix='', suffix='') -%} + {{ return(adapter.dispatch('star', 'dbt_utils')(from, relation_alias, except, prefix, suffix)) }} {% endmacro %} -{% macro default__star(from, relation_alias=False, except=[]) -%} +{% macro default__star(from, relation_alias=False, except=[], prefix='', suffix='') -%} {%- do dbt_utils._is_relation(from, 'star') -%} {%- do dbt_utils._is_ephemeral(from, 'star') -%} @@ -24,7 +24,7 @@ {%- for col in include_cols %} - {%- if relation_alias %}{{ relation_alias }}.{% else %}{%- endif -%}{{ adapter.quote(col)|trim }} + {%- if relation_alias %}{{ relation_alias }}.{% else %}{%- endif -%}{{ adapter.quote(col)|trim }} as {{ adapter.quote(prefix ~ col ~ suffix)|trim }} {%- if not loop.last %},{{ '\n ' }}{% endif %} {%- endfor -%} From baf4053543ca11953de7ec0a75e2016da716bce8 Mon Sep 17 00:00:00 2001 From: Joel Labes Date: Fri, 3 Dec 2021 10:56:15 +1300 Subject: [PATCH 5/5] dbt v1 compat (#453) * dbt 0.7.4 release (#441) * Update require-dbt-version to be 1.0 * Fix SQL 42000 on Exasol (#420) " SQL-Error [42000]: syntax error, unexpected '*' " If you specify the * in the unioned with their respectiv names .* you do not receive the SQL Error posted above. This should not inflict any further problems since it is redundant for most DBs. * Minor readme link fixes (#431) * minor readme link fixes * changelog addition Co-authored-by: Joel Labes * 0.7.4 changelog (#432) * Update CHANGELOG.md * Note branch name change * use `limit_zero` macro instead of `limit 0` (#437) * Utils 0.7.4b1 (#433) * Update require-dbt-version to be 1.0 * Fix SQL 42000 on Exasol (#420) " SQL-Error [42000]: syntax error, unexpected '*' " If you specify the * in the unioned with their respectiv names .* you do not receive the SQL Error posted above. This should not inflict any further problems since it is redundant for most DBs. * Minor readme link fixes (#431) * minor readme link fixes * changelog addition Co-authored-by: Joel Labes * 0.7.4 changelog (#432) * Update CHANGELOG.md * Note branch name change Co-authored-by: Timo Kruth Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * standard convention * Update integration_tests/tests/jinja_helpers/test_slugify.sql Taking the liberty of committing on your behalf so that the CI job starts again * Change limit_zero to be a macro Co-authored-by: Joel Labes Co-authored-by: Timo Kruth Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> * Add col_name alias to else state too (#437) * Remove extra semicolon in `insert_by_period` materialization (#439) * Remove extra semicolon in `insert_by_period` materialization. `create_table_as()` generates a SQL statement that already ends with a semicolon, so the extra semicolon after a `create_table_as()` call in the `insert_by_period` materialization ends up being an empty SQL statement, and at least when using Snowflake this causes the dbt run to fail with a "cannot unpack non-iterable NoneType object" error. * Update changelog for PR 439. * Use the relation object passed into get_column_values, instead of making our own (#440) * Use the relation object passed into get_column_values, instead of making our own * Rename variables in get column value test to be clearer * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Timo Kruth Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Co-authored-by: Anders Co-authored-by: Sean Rose * Regression: Correctly handle missing relations in get_column_values (#448) * Create integration test for a dropped relation * Update get_column_values.sql * Swap out adapter call for a good old fashioned drop table * Add missing curlies * what person wrote this code :/ (it was me) * wrap values in quotes * GOOD * bigquery compat (they don't like except) * Backport android url changes from #426 (#452) * Update CHANGELOG.md * Change require-dbt-version, update dbt_project.yml for integration tests proj * Upgrade python version in CI, improve drop relation integration test * Clarify version pinning * Drop support for release candidates of 1.0.0 Co-authored-by: Timo Kruth Co-authored-by: Joe Markiewicz <74217849+fivetran-joemarkiewicz@users.noreply.github.com> Co-authored-by: Anders Co-authored-by: Sean Rose --- .circleci/config.yml | 8 ++++---- CHANGELOG.md | 2 +- dbt_project.yml | 6 +++--- integration_tests/dbt_project.yml | 10 +++++----- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/.circleci/config.yml b/.circleci/config.yml index 8869c8bb..2432273b 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -5,7 +5,7 @@ jobs: integration-postgres: docker: - - image: circleci/python:3.6.13-stretch + - image: cimg/python:3.9.9 - image: circleci/postgres:9.6.5-alpine-ram steps: @@ -24,7 +24,7 @@ jobs: integration-redshift: docker: - - image: circleci/python:3.6.3-stretch + - image: cimg/python:3.9.9 steps: - checkout - run: @@ -35,7 +35,7 @@ jobs: integration-snowflake: docker: - - image: circleci/python:3.6.3-stretch + - image: cimg/python:3.9.9 steps: - checkout - run: @@ -48,7 +48,7 @@ jobs: environment: BIGQUERY_SERVICE_KEY_PATH: "/home/circleci/bigquery-service-key.json" docker: - - image: circleci/python:3.6.3-stretch + - image: cimg/python:3.9.9 steps: - checkout - run: diff --git a/CHANGELOG.md b/CHANGELOG.md index 75833b7b..6ad5e3e0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,6 @@ # dbt-utils v0.8.0 ## 🚨 Breaking changes -- dbt ONE POINT OH is here! This version of dbt-utils requires `dbt-core ~=1.0.0` (any minor and patch version of v1), which means far less need for compatibility releases in the future. +- dbt ONE POINT OH is here! This version of dbt-utils requires _any_ version (minor and patch) of v1, which means far less need for compatibility releases in the future. - The partition column in the `mutually_exclusive_ranges` test is now always called `partition_by_col`. This enables compatibility with `--store-failures` when multiple columns are concatenated together. If you have models built on top of the failures table, update them to reflect the new column name. ([#423](https://github.com/dbt-labs/dbt-utils/issues/423), [#430](https://github.com/dbt-labs/dbt-utils/pull/430)) ## Contributors: diff --git a/dbt_project.yml b/dbt_project.yml index aaaf9d49..833501d8 100644 --- a/dbt_project.yml +++ b/dbt_project.yml @@ -1,11 +1,11 @@ name: 'dbt_utils' -version: '0.7.0' +version: '0.1.0' -require-dbt-version: [">=0.20.0", "<=1.0.0"] +require-dbt-version: [">=1.0.0", "<2.0.0"] config-version: 2 target-path: "target" -clean-targets: ["target", "dbt_modules"] +clean-targets: ["target", "dbt_modules", "dbt_packages"] macro-paths: ["macros"] log-path: "logs" diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml index 9631a8c4..bb6d147b 100644 --- a/integration_tests/dbt_project.yml +++ b/integration_tests/dbt_project.yml @@ -8,16 +8,17 @@ profile: 'integration_tests' config-version: 2 -source-paths: ["models"] +model-paths: ["models"] analysis-paths: ["analysis"] test-paths: ["tests"] -data-paths: ["data"] +seed-paths: ["data"] macro-paths: ["macros"] target-path: "target" # directory which will store compiled SQL files clean-targets: # directories to be removed by `dbt clean` - "target" - "dbt_modules" + - "dbt_packages" dispatch: - macro_namespace: 'dbt_utils' @@ -56,9 +57,8 @@ seeds: +schema: events data_get_column_values_dropped: - # TODO: Ideally this would use the adapter, but it gives a "Tried to drop relation, but its type is null" error. - #+post-hook: "{% do adapter.drop_relation(this) %}" - +post-hook: "drop table {{ this }}" + # this.incorporate() to hardcode the node's type as otherwise dbt doesn't know it yet + +post-hook: "{% do adapter.drop_relation(this.incorporate(type='table')) %}" schema_tests: