make patch update to account for upstream changes #182

fivetran-reneeli · 2024-12-30T21:09:23Z

PR Overview

This PR will address the following Issue/Feature:
Internal ticket where empty tables using Redshift will ignore explicit data casts and cast everything as varchar by default, which presents problems downstreams in transformations. By creating a table with 1 row with all null records for each empty source table, this ensures that Redshift keeps the explicit datatype casts.

This PR will result in the following new package version: v0.20.0

Not a breaking change in itself, but needed to upgrade the downstream transforms package so it can be picked up by Quickstart.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Upstream Under-the-Hood Updates from `zendesk_source` Package

(Affects Redshift only) Updates the union_zendesk_connections macro to use a limit 1 instead of limit 0 for empty tables.
- When a table is empty, Redshift ignores explicit data casts and will materialize every column as a varchar. Redshift users may experience errors in downstream transformations as a consequence.
- For each staging model, if the source table is not found, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-null row for Redshift destinations. The 1 row will ensure that Redshift will respect the package's datatype casts.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

dbt run –full-refresh && dbt test
dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

[na] The appropriate issue has been linked, tagged, and properly assigned
All necessary documentation and version upgrades have been applied
[na] docs were regenerated (unless this PR does not include any code or yml updates)
BuildKite integration tests are passing
Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

Recreate the issue using an empty schema on Redshift and running the transforms package on prod:

Running on dev:

(although the constant expressions error is still present)

If you had to summarize this PR in an emoji, which would it be?

💃

fivetran-jamie

Same as source -- looks pretty good but can you regen the docs to include the macro update?

And my question regarding the constant expression error stands -- was this only coming up when ticket_field_history was missing?

fivetran-jamie · 2025-01-07T21:30:40Z

packages.yml

+  # - package: fivetran/zendesk_source
+  #   version: [">=0.14.0", "<0.15.0"]
  - package: calogica/dbt_date
-    version: [">=0.9.0", "<1.0.0"]
+    version: [">=0.9.0", "<1.0.0"]
+
+  - git: https://github.com/fivetran/dbt_zendesk_source.git
+    revision: bugfix/redshift_empty_tables
+    warn-unpinned: false


reminder to switch to v0.15.* before merging

fivetran-reneeli · 2025-01-08T16:48:55Z

Thanks @fivetran-jamie ! Regenerated the docs. Let's keep correspondence re: ticket_field_history in the source

fivetran-jamie

LGTM

fivetran-avinash · 2025-01-09T22:06:57Z

CHANGELOG.md

@@ -1,3 +1,10 @@
+# dbt_zendesk v0.20.0
+
+## [Upstream Under-the-Hood Updates](https://github.com/fivetran/dbt_zendesk_source/blob/main/CHANGELOG.md) from `zendesk_source` Package


Upstream and "from zendesk_source Package" is redundant. I'd suggest keeping one or the other but not both.

I'd link directly to the release notes for the source package rather than the CHANGELOG, as the CHANGELOG will change over time. Either that or the release hyperlink (https://github.com/fivetran/dbt_zendesk/blob/main/CHANGELOG.md#dbt_zendesk_source-v0200) I think will be the link? Might want to check.

thanks! updated

fivetran-avinash · 2025-01-09T22:10:08Z

README.md

@@ -1,3 +1,5 @@
+# Zendesk Support Modeling dbt Package ([Docs](https://fivetran.github.io/dbt_zendesk/))


Should we move the H1 heading in the dbt_zendesk_source package too? No worries if it's out of scope.

yes good reminder

fivetran-avinash · 2025-01-09T22:13:33Z

packages.yml

@@ -1,5 +1,10 @@
 packages:
-  - package: fivetran/zendesk_source
-    version: [">=0.14.0", "<0.15.0"]
+  # - package: fivetran/zendesk_source


Reminder to switch deps before merging.

fivetran-avinash · 2025-01-09T22:14:22Z

CHANGELOG.md

+- (Affects Redshift only) Updates the `union_zendesk_connections` macro to use a limit 1 instead of limit 0 for empty tables.
+  - When a table is empty, Redshift ignores explicit data casts and will materialize every column as a `varchar`. Redshift users may experience errors in downstream transformations as a consequence.
+  - For each staging model, if the source table is not found, the package will create a empty table with 0 rows for non-Redshift warehouses and a table with 1 all-`null` row for Redshift destinations. The 1 row will ensure that Redshift will respect the package's datatype casts.
+


Should we create a Documentation section to mention the movement of the H1 heading?

Hmm I don't see other release notes in other packages mentioning this, and it's so minor I don't think it would have much impact?

Oh I guess I just added them in my recent ones. https://github.com/fivetran/dbt_zuora/releases/tag/v0.3.1

But it's not a big deal either way.

fivetran-avinash

A few comments and suggestions to consider, but otherwise approved pending the current quickstart.yml issue.

fivetran-reneeli added 3 commits December 30, 2024 14:56

make patch update to account for upstream changes

41b5ce3

temp packages deps

1dd12c5

readme versioning

c865bef

fivetran-reneeli requested a review from fivetran-jamie January 3, 2025 16:05

fivetran-jamie reviewed Jan 7, 2025

View reviewed changes

fivetran-reneeli and others added 4 commits January 8, 2025 10:55

Merge branch 'main' into bugfix/redshift_empty_tables

eb00bcb

packages deps

e224f41

docs

8e43d9d

new schema

6c07eac

fivetran-jamie approved these changes Jan 8, 2025

View reviewed changes

fivetran-reneeli added 3 commits January 9, 2025 15:37

merge main

cbad986

docs

2b6b2c1

schema

e47da2b

fivetran-avinash reviewed Jan 9, 2025

View reviewed changes

fivetran-avinash approved these changes Jan 9, 2025

View reviewed changes

fivetran-reneeli added 3 commits January 10, 2025 15:46

doc updates

ef99d1c

test again

0083a27

change deps, update docs, add header change to cl

bb47018

fivetran-reneeli merged commit 9751431 into main Jan 13, 2025
9 checks passed

fivetran-reneeli deleted the bugfix/redshift_empty_tables branch January 13, 2025 17:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make patch update to account for upstream changes #182

make patch update to account for upstream changes #182

fivetran-reneeli commented Dec 30, 2024 •

edited

Loading

fivetran-jamie left a comment

fivetran-jamie Jan 7, 2025

fivetran-reneeli commented Jan 8, 2025

fivetran-jamie left a comment

fivetran-avinash Jan 9, 2025 •

edited

Loading

fivetran-reneeli Jan 10, 2025

fivetran-avinash Jan 9, 2025

fivetran-reneeli Jan 10, 2025

fivetran-reneeli Jan 10, 2025

fivetran-avinash Jan 9, 2025

fivetran-avinash Jan 9, 2025

fivetran-reneeli Jan 10, 2025

fivetran-avinash Jan 10, 2025

fivetran-avinash left a comment •

edited

Loading

		@@ -1,3 +1,10 @@
		# dbt_zendesk v0.20.0

		## [Upstream Under-the-Hood Updates](https://github.com/fivetran/dbt_zendesk_source/blob/main/CHANGELOG.md) from `zendesk_source` Package

		@@ -1,3 +1,5 @@
		# Zendesk Support Modeling dbt Package ([Docs](https://fivetran.github.io/dbt_zendesk/))

make patch update to account for upstream changes #182

make patch update to account for upstream changes #182

Conversation

fivetran-reneeli commented Dec 30, 2024 • edited Loading

PR Overview

Upstream Under-the-Hood Updates from zendesk_source Package

PR Checklist

Basic Validation

Detailed Validation

If you had to summarize this PR in an emoji, which would it be?

fivetran-jamie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-reneeli commented Jan 8, 2025

fivetran-jamie left a comment

Choose a reason for hiding this comment

fivetran-avinash Jan 9, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fivetran-avinash left a comment • edited Loading

Choose a reason for hiding this comment

fivetran-reneeli commented Dec 30, 2024 •

edited

Loading

Upstream Under-the-Hood Updates from `zendesk_source` Package

fivetran-avinash Jan 9, 2025 •

edited

Loading

fivetran-avinash left a comment •

edited

Loading