Feature/historical schedules (#171)

* send to bk * try this out * first try * use mcro * Testing * puhs * redshift * try nullif * feature/historical-schedules * rework schedule_history * update unnest logic * update schedule_history * complete schedule_history * revise holidays * add macro * typeo * updates * updates * updates * updates * try out in buildkite * remove schedule days during holiday * revise * add config * allow disable holidays * streamline bk run * adjust for multiple holidays in a week * add casting * fixes * fix multiyear schedules * add longer holiday support * revert multiweek * adjust multiweek * account for non sunday week starts * updates * update weeks spanned calc * update naming * update to dbt_date * update comments * update to dbt date weekstart * modernize calendar spine * make sure we're working with strings when replacing * let's see if bk works * postgres? * i think schedule history may actually be workingggg * get ready to merge into catherines branch * postgres revert * add comments * redshift fixes * updates * validation update * split models * update ymls * revise scchedule groups * updates * add inline comments * update decision log * update changelog * regen docs * fix yml * update changelog * address review comments and regen docs * address review comments * Apply suggestions from code review Co-authored-by: Joe Markiewicz <[email protected]> * address review comments * Update packages.yml * release review updates --------- Co-authored-by: Jamie Rodriguez <[email protected]> Co-authored-by: Joe Markiewicz <[email protected]>
fivetran · Oct 10, 2024 · a9be770 · a9be770
1 parent 0b73c19
commit a9be770
Show file tree

Hide file tree

Showing 32 changed files with 1,053 additions and 386 deletions.
diff --git a/.buildkite/scripts/run_models.sh b/.buildkite/scripts/run_models.sh
@@ -18,8 +18,13 @@ cd integration_tests
 dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
+dbt run --target "$db"
 dbt test --target "$db"
 dbt run --vars '{zendesk__unstructured_enabled: true, using_schedules: false, using_domain_names: false, using_user_tags: false, using_ticket_form_history: false, using_organization_tags: false}' --target "$db" --full-refresh
 dbt test --target "$db"
+dbt run --vars '{using_schedule_histories: true, using_holidays: false}' --target "$db" --full-refresh
+dbt test --target "$db"
+dbt run --vars '{using_schedule_histories: true, using_holidays: true}' --target "$db" --full-refresh
+dbt test --target "$db"
 
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"
diff --git a/.quickstart/quickstart.yml b/.quickstart/quickstart.yml
@@ -6,7 +6,6 @@ dbt_versions: ">=1.3.0 <2.0.0"
 table_variables:
   using_schedules:
     - daylight_time
-    - schedule_holiday
     - schedule
     - time_zone
   using_domain_names:
@@ -17,6 +16,10 @@ table_variables:
     - ticket_form_history
   using_organization_tags:
     - organization_tag
+  using_schedule_histories:
+    - audit_log
+  using_holidays:
+    - schedule_holiday
 
 destination_configurations:
   databricks:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,5 +1,52 @@
-# dbt_zendesk v0.17.0
+# dbt_zendesk v0.18.0
+[PR #171](https://github.com/fivetran/dbt_zendesk/pull/171) includes the following changes:
+
+## Breaking Changes (Full refresh required after upgrading)
+### Schedule Change Support
+- Support for schedule changes has been added. This feature is disabled by default since most users do not sync the required source `audit_table`. To enable this feature set the variable `using_schedule_histories` to `true` in your `dbt_project.yml`:
+```yml
+vars:
+  using_schedule_histories: true
+```
+  - Schedule changes can now be extracted directly from the audit log, providing a view of schedule modifications over time.
+  - The `int_zendesk__schedule_spine` model is now able to incorporate these schedule changes, making it possible for downstream models to reflect the most up-to-date schedule data.
+    - Note this is only in effect when `using_schedule_histories` is true.
+  - This improves granularity for Zendesk metrics related to agent availability, SLA tracking, and time-based performance analysis.
+#### dbt_zendesk_source changes (see the [Release Notes](https://github.com/fivetran/dbt_zendesk_source/releases/tag/v0.13.0) for more details)
+- Introduced the `stg_zendesk__audit_log` table for capturing schedule changes from Zendesk's audit log.
+  - This model is disabled by default, to enable it set variable `using_schedule_histories` to `true` in `dbt_project.yml`.
+
+## New Features
+- Holiday support: Users can now choose to disable holiday tracking, while continuing to use schedules, by setting variable `using_holidays` to `false` in `dbt_project.yml`.
+- New intermediate models have been introduced to streamline both the readability and maintainability:
+  - [`int_zendesk__timezone_daylight`](https://github.com/fivetran/dbt_zendesk/blob/main/models/utils/int_zendesk__timezone_daylight.sql): A utility model that maintains a record of daylight savings adjustments for each time zone.
+    - materialization: ephemeral
+  - [`int_zendesk__schedule_history`](https://github.com/fivetran/dbt_zendesk/blob/main/models/intermediate/int_zendesk__schedule_history.sql): Captures a full history of schedule changes for each `schedule_id`.
+    - materialization: table (if enabled)
+  - [`int_zendesk__schedule_timezones`](https://github.com/fivetran/dbt_zendesk/blob/main/models/intermediate/int_zendesk__schedule_timezones.sql): Merges schedule history with time zone shifts.
+    - materialization: ephemeral
+  - [`int_zendesk__schedule_holiday`](https://github.com/fivetran/dbt_zendesk/blob/main/models/intermediate/int_zendesk__schedule_holiday.sql): Identifies and calculates holiday periods for each schedule.
+    - materialization: ephemeral
+- Rebuilt logic in [`int_zendesk__schedule_spine`](https://github.com/fivetran/dbt_zendesk/blob/main/models/intermediate/int_zendesk__schedule_spine.sql) to consolidate updates from the new intermediate models.
+#### dbt_zendesk_source changes (see the [Release Notes](https://github.com/fivetran/dbt_zendesk_source/releases/tag/v0.13.0) for more details)
+- Updated the `stg_zendesk__schedule_holidays` model to allow users to disable holiday processing by setting variable `using_holidays` to `false`.
+
+## Bug Fixes
+- Resolved a bug in the `int_zendesk__schedule_spine` model where users experienced large gaps in non-holiday periods. The updated logic addresses this issue.
 
+## Decision log
+- Added the following [DECISIONLOG](https://github.com/fivetran/dbt_zendesk/blob/main/DECISIONLOG.md) entries:
+  - Entry addressing how multiple schedule changes in a single day are handled. Only the last change of the day is captured to align with day-based downstream logic.
+  - Entry to clarify backfilling of schedule history. The most recent schedule is sourced from `stg_zendesk__schedule`, while historical changes are managed separately, allowing users to disable the history feature if needed.
+
+## Under the Hood
+- Replaced instances of `dbt.date_trunc` with `dbt_date.week_start` to standardize week start dates to Sunday across all warehouses, since our schedule logic relies on consistent weeks.
+- Replaced the deprecated `dbt.current_timestamp_backcompat()` function with `dbt.current_timestamp()` to ensure all timestamps are captured in UTC.
+- Added seed data for `audit_log` to enhance integration testing capabilities.
+- Introduced new helper macros, `clean_data` and `regex_extract`, to process complex text of the schedule changes extracted from audit logs.
+- Updated `int_zendesk__calendar_spine` logic to prevent errors during compilation before the first full run, ensuring a smoother development experience.
+
+# dbt_zendesk v0.17.0
 ## New model ([#161](https://github.com/fivetran/dbt_zendesk/pull/161))
 - Addition of the `zendesk__document` model, designed to structure Zendesk textual data for vectorization and integration into NLP workflows. The model outputs a table with:
   - `document_id`: Corresponding to the `ticket_id`

diff --git a/DECISIONLOG.md b/DECISIONLOG.md
@@ -1,12 +1,14 @@
 # Decision Log
 
-## Tracking Ticket SLA Policies Into the Future
-In our models we generate a future time series for ticket SLA policies. This is limited to a year to maintain performance. 
+## Schedule History
+### Handling Multiple Schedule Changes in a Day
+While integrating schedule changes from the audit_log source, we observed that multiple changes can occur on the same day, often when users are still finalizing a schedule. To maintain clarity and align with our day-based downstream logic, we decided to capture only the last change made on any given day. If this approach proves insufficient for your use case, please submit a [feature request](https://github.com/fivetran/dbt_zendesk/issues/new/choose) for enabling support for multiple changes within a single day.
 
-## No Historical Schedule Reference
-At the current moment the Fivetran Zendesk Support connector does not contain historical data of schedules. This means if a schedule is created in the Zendesk Support UI and remains untouched for years, but then is adjusted in the current month you will see the data synced in the raw `schedule` table to reflect the current adjusted schedule. As a result the raw data will lose all historical reference of what this schedule range was previously.
+### Backfilling the Schedule History
+Although the schedule history extracted from the audit log includes the most recent schedule, we exclude it in the `int_zendesk__schedule_history` model. Instead, we rely on the schedule from `stg_zendesk__schedule`, since it represents the live schedule. This approach also allows users who are not using schedule histories to easily disable the history feature. We join the live schedule with the schedule history model and bridge the valid_from and valid_until dates to maintain consistency.
 
-Therefore, if you are leveraging the `using_schedule` variable as `true` to replicate business hour metrics this data model will only have a reference to the current range of any given schedule. This means tickets from the previous two years that were leveraging the __old__ schedule will not be reported as using the __new__ schedule. If this data limitation is a concern to you, we recommend opening a [Fivetran Support Feature Request](https://support.fivetran.com/hc/en-us/community/topics/360001909373-Feature-Requests?sort_by=votes) to enhance the Zendesk Support connector to include historical schedule data.
+## Tracking Ticket SLA Policies Into the Future
+In our models we generate a future time series for ticket SLA policies. This is limited to a year to maintain performance. 
 
 ## Zendesk Support First Reply Time SLA Opinionated Logic
 The logic for `first_reply_time` breach/achievement metrics within the `zendesk__ticket_metrics` and `zendesk__sla_policies` models are structured on the Zendesk Support definition of [first reply time SLA events](https://support.zendesk.com/hc/en-us/articles/4408821871642-Understanding-ticket-reply-time?page=2#topic_jvw_nqd_1hb). For example, this data model calculates first reply time to be the duration of time (business or calendar) between the creation of the ticket and the first public comment from either an `agent` or `admin`. This holds true regardless of when the first reply time SLA was applied to the ticket.

diff --git a/README.md b/README.md
@@ -65,7 +65,7 @@ Include the following zendesk package version in your `packages.yml` file:
 ```yml
 packages:
   - package: fivetran/zendesk
-    version: [">=0.17.0", "<0.18.0"]
+    version: [">=0.18.0", "<0.19.0"]
 
 ```
 > **Note**: Do not include the Zendesk Support source package. The Zendesk Support transform package already has a dependency on the source in its own `packages.yml` file.
@@ -79,11 +79,13 @@ vars:
     zendesk_schema: your_schema_name 
 ```
 
-### Step 4: Disable models for non-existent sources
-This package takes into consideration that not every Zendesk Support account utilizes the `schedule`, `schedule_holiday`, `ticket_schedule` `daylight_time`, `time_zone`, `domain_name`, `user_tag`, `organization_tag`, or `ticket_form_history` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true`. Add variables for only the tables you want to disable:
+### Step 4: Enable/Disable models for non-existent sources
+This package takes into consideration that not every Zendesk Support account utilizes the `schedule`, `schedule_holiday`, `ticket_schedule`, `daylight_time`, `time_zone`, `audit_log`, `domain_name`, `user_tag`, `organization_tag`, or `ticket_form_history` features, and allows you to disable the corresponding functionality. By default, all variables' values are assumed to be `true`, except for `using_schedule_histories`. Add variables for only the tables you want to enable/disable:
 ```yml
 vars:
-    using_schedules:            False         #Disable if you are not using schedules
+    using_schedule_histories:   True          #Enable if you are using audit_logs for schedule histories
+    using_schedules:            False         #Disable if you are not using schedules, which requires source tables ticket_schedule, daylight_time, and time_zone  
+    using_holidays:             False         #Disable if you are not using schedule_holidays for holidays
     using_domain_names:         False         #Disable if you are not using domain names
     using_user_tags:            False         #Disable if you are not using user tags
     using_ticket_form_history:  False         #Disable if you are not using ticket form history
@@ -231,7 +233,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/zendesk_source
-      version: [">=0.12.0", "<0.13.0"]
+      version: [">=0.13.0", "<0.14.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,6 +1,5 @@
 name: 'zendesk'
-version: '0.17.0'
-
+version: '0.18.0'
 
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
@@ -14,6 +13,10 @@ models:
     intermediate:
       +schema: zendesk_intermediate
       +materialized: table
+      int_zendesk__schedule_timezones:
+        +materialized: ephemeral
+      int_zendesk__schedule_holiday:
+        +materialized: ephemeral
     reply_times:
       +materialized: ephemeral
     resolution_times:
@@ -33,23 +36,24 @@ vars:
   zendesk:
     ticket_field_history_columns: ['assignee_id', 'status', 'priority']
     ticket_field_history_updater_columns: []
-    group: "{{ ref('stg_zendesk__group') }}"
+    audit_log: "{{ ref('stg_zendesk__audit_log') }}"
     brand: "{{ ref('stg_zendesk__brand') }}"
+    daylight_time: "{{ ref('stg_zendesk__daylight_time') }}"
     domain_name: "{{ ref('stg_zendesk__domain_name') }}"
+    field_history: "{{ ref('stg_zendesk__ticket_field_history') }}"
+    group: "{{ ref('stg_zendesk__group') }}"
     organization_tag: "{{ ref('stg_zendesk__organization_tag') }}"
     organization: "{{ ref('stg_zendesk__organization') }}"
-    schedule: "{{ ref('stg_zendesk__schedule') }}"
     schedule_holiday: "{{ ref('stg_zendesk__schedule_holiday') }}"
-    ticket: "{{ ref('stg_zendesk__ticket') }}"
-    ticket_form_history: "{{ ref('stg_zendesk__ticket_form_history') }}"
+    schedule: "{{ ref('stg_zendesk__schedule') }}"
     ticket_comment: "{{ ref('stg_zendesk__ticket_comment') }}"
-    field_history: "{{ ref('stg_zendesk__ticket_field_history') }}"
+    ticket_form_history: "{{ ref('stg_zendesk__ticket_form_history') }}"
     ticket_schedule: "{{ ref('stg_zendesk__ticket_schedule') }}"
     ticket_tag: "{{ ref('stg_zendesk__ticket_tag') }}"
+    ticket: "{{ ref('stg_zendesk__ticket') }}"
+    time_zone: "{{ ref('stg_zendesk__time_zone') }}"
     user_tag: "{{ ref('stg_zendesk__user_tag') }}"
     user: "{{ ref('stg_zendesk__user') }}"
-    daylight_time: "{{ ref('stg_zendesk__daylight_time') }}"
-    time_zone: "{{ ref('stg_zendesk__time_zone') }}"
     using_schedules: true
     using_domain_names: true
     using_user_tags: true

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'zendesk_integration_tests'
-version: '0.17.0'
+version: '0.18.0'
 
 profile: 'integration_tests'
 
@@ -25,27 +25,26 @@ vars:
     zendesk_organization_tag_identifier:       "organization_tag_data"
     zendesk_user_identifier:                   "user_data"
     zendesk_user_tag_identifier:               "user_tag_data"
+    zendesk_audit_log_identifier:              "audit_log_data"
 
   ## Uncomment for docs generation
   # zendesk__unstructured_enabled: True
+  # using_schedule_histories: True
 
   ## For validation testing. To be commented out before release.
   # zendesk_schema: zendesk_test_env
   # using_domain_names: false
   # using_user_tags: false
   # using_organization_tags: false
-  # fivetran_integrity_sla_first_reply_time_exclusion_tickets: (1,56,80)
-  # fivetran_consistency_ticket_metrics_exclusion_tickets: (11092,11093,11094)
-  # fivetran_integrity_sla_count_match_tickets: (76)
+  # fivetran_integrity_sla_metric_parity_exclusion_tickets: (56,80)
+  # fivetran_integrity_sla_first_reply_time_exclusion_tickets: (56,80)
 
 models:
   +schema: "zendesk_{{ var('directed_schema','dev') }}"
 
 seeds:
   +quote_columns: "{{ true if target.type == 'redshift' else false }}"
   zendesk_integration_tests:
-    +column_types:
-      _fivetran_synced: timestamp
     +column_types:
       _fivetran_synced: timestamp
     group_data: