Merge pull request #43 from fivetran/feature/tag-former-agents

Add var to persist former-agents' roles for SLA metrics

.github/PULL_REQUEST_TEMPLATE/maintainer_pull_request_template.md

@@ -11,15 +11,20 @@
 ### Basic Validation
 Please acknowledge that you have successfully performed the following commands locally:
 - [ ] dbt run –full-refresh && dbt test
-- [ ] dbt run (if incremental models are present)
+- [ ] dbt run (if incremental models are present) && dbt test
 
 Before marking this PR as "ready for review" the following have been applied:
 - [ ] The appropriate issue has been linked, tagged, and properly assigned
 - [ ] All necessary documentation and version upgrades have been applied
+    <!--- Be sure to update the package version in the dbt_project.yml, integration_tests/dbt_project.yml, and README if necessary. -->
 - [ ] docs were regenerated (unless this PR does not include any code or yml updates)
 - [ ] BuildKite integration tests are passing
 - [ ] Detailed validation steps have been provided below
 
 ### Detailed Validation
 Please share any and all of your validation steps:
-<!--- Provide the steps you took to validate your changes below. -->
+<!--- Provide the steps you took to validate your changes below. -->
+
+### If you had to summarize this PR in an emoji, which would it be?
+<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908)  --> 
+:dancer:

.github/workflows/auto-release.yml

@@ -0,0 +1,13 @@
+name: 'auto release'
+on:
+  pull_request:
+    types:
+      - closed
+    branches:
+      - main
+
+jobs:
+  call-workflow-passing-data:
+    if: github.event.pull_request.merged
+    uses: fivetran/dbt_package_automations/.github/workflows/auto-release.yml@main
+    secrets: inherit

CHANGELOG.md

@@ -1,3 +1,22 @@
+# dbt_zendesk_source v0.10.1
+[PR #43](https://github.com/fivetran/dbt_zendesk_source/pull/43) introduces the following updates:
+
+## Feature Updates
+- Added the `internal_user_criteria` variable, which can be used to mark internal users whose `USER.role` may have changed from `agent` to `end-user` after they left your organization. This variable accepts SQL that may reference any non-custom field in `USER`, and it will be wrapped in a `case when` statement in the `stg_zendesk__user` model.
+  - Example usage:
+```yml
+# dbt_project.yml
+vars:
+  zendesk_source:
+    internal_user_criteria: "lower(email) like '%@fivetran.com' or external_id = '12345' or name in ('Garrett', 'Alfredo')" # can reference any non-custom field in USER
+```
+  - Output: In `stg_zendesk__user`, users who match your criteria and have a role of `end-user` will have their role switched to `agent`. This will ensure that downstream SLA metrics are appropriately calculated.
+
+## Under the Hood
+- Updated the way we dynamically disable sources. Previously, we used a custom `meta.is_enabled` flag, but, since we added this, dbt-core introduced a native `config.enabled` attribute.  We have opted to use the dbt-native config instead.
+- Updated the pull request [templates](/.github).
+- Included auto-releaser GitHub Actions workflow to automate future releases.
+
 # dbt_zendesk_source v0.10.0
 [PR #42](https://github.com/fivetran/dbt_zendesk_source/pull/42) introduces the following updates: 
 

README.md

@@ -66,15 +66,27 @@ vars:
 ```
 
 ## (Optional) Step 5: Additional configurations
-<details><summary>Expand to view configurations</summary>
-
+
 ### Add passthrough columns
 This package includes all source columns defined in the staging models. However, the `stg_zendesk__ticket` model allows for additional columns to be added using a pass-through column variable. This is extremely useful if you'd like to include custom fields to the package.
 ```yml
 vars:
   zendesk__ticket_passthrough_columns: [account_custom_field_1, account_custom_field_2]
 ```
 
+### Mark Former Internal Users as Agents
+If a team member leaves your organization and their internal account is deactivated, their `USER.role` will switch from `agent` or `admin` to `end-user`. This will skew historical ticket SLA metrics, as we calculate reply times and other metrics based on `agent` or `admin` activity only.
+
+To persist the integrity of historical ticket SLAs and mark these former team members as agents, provide the `internal_user_criteria` variable with a SQL clause to identify them, based on fields in the `USER` table. This SQL will be wrapped in a `case when` statement in the `stg_zendesk__user` model.
+
+Example usage:
+```yml
+# dbt_project.yml
+vars:
+  zendesk_source:
+    internal_user_criteria: "lower(email) like '%@fivetran.com' or external_id = '12345' or name in ('Garrett', 'Alfredo')" # can reference any non-custom field in USER
+```
+
 ### Change the build schema
 By default, this package builds the zendesk staging models within a schema titled (`<target_schema>` + `_zendesk_source`) in your target database. If this is not where you would like your Zendesk staging data to be written to, add the following configuration to your root `dbt_project.yml` file:
 
@@ -102,8 +114,7 @@ In this package, this would apply to the `GROUP` source. If you are receiving er
 vars:
     zendesk_group_identifier: "Group" # as an example, must include the double-quotes and correct case!
 ```  
-
-</details>
+
 
 ## (Optional) Step 6: Orchestrate your models with Fivetran Transformations for dbt Core™
 <details><summary>Expand to view details</summary>

dbt_project.yml

@@ -1,6 +1,6 @@
 config-version: 2
 name: 'zendesk_source'
-version: '0.10.0'
+version: '0.10.1'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   zendesk_source:

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'zendesk_source_integration_tests'
-version: '0.10.0'
+version: '0.10.1'
 
 profile: 'integration_tests'
 

models/src_zendesk.yml

@@ -91,8 +91,8 @@ sources:
       - name: domain_name
         identifier: "{{ var('zendesk_domain_name_identifier', 'domain_name')}}"
         description: Domain names associated with an organization. An organization may have multiple domain names.
-        meta:
-          is_enabled: "{{ var('using_domain_names', true) }}"
+        config:
+          enabled: "{{ var('using_domain_names', true) }}"
         columns:
           - name: organization_id
             description: Reference to the organization
@@ -119,8 +119,8 @@ sources:
       - name: organization_tag
         identifier: "{{ var('zendesk_organization_tag_identifier', 'organization_tag') }}"
         description: The tags associated with an organization. An organization may have multiple tags.
-        meta:
-          is_enabled: "{{ var('using_organization_tags', true) }}"
+        config:
+          enabled: "{{ var('using_organization_tags', true) }}"
         columns:
           - name: organization_id
             description: Reference to the organization
@@ -194,8 +194,8 @@ sources:
       - name: user_tag
         identifier: "{{ var('zendesk_user_tag_identifier', 'user_tag') }}"
         description: Table containing all tags associated with a user. Only present if your account has user tagging enabled.
-        meta:
-          is_enabled: "{{ var('using_user_tags', true) }}"
+        config:
+          enabled: "{{ var('using_user_tags', true) }}"
         columns:
           - name: user_id
             description: Reference to the user
@@ -230,8 +230,8 @@ sources:
         identifier: "{{ var('zendesk_schedule_identifier', 'schedule') }}"
         description: The support schedules created with different business hours and holidays. 
         freshness: null
-        meta:
-          is_enabled: "{{ var('using_schedules', true) }}"
+        config:
+          enabled: "{{ var('using_schedules', true) }}"
         columns: 
           - name: id
             description: ID automatically assigned to the schedule upon creation
@@ -261,8 +261,8 @@ sources:
       - name: ticket_form_history
         identifier: "{{ var('zendesk_ticket_form_history_identifier', 'ticket_form_history') }}"
         description: Ticket forms allow an admin to define a subset of ticket fields for display to both agents and end users.
-        meta:
-          is_enabled: "{{ var('using_ticket_form_history', true) }}"
+        config:
+          enabled: "{{ var('using_ticket_form_history', true) }}"
         columns:
           - name: id
             description: Automatically assigned when creating ticket form
@@ -336,8 +336,8 @@ sources:
         identifier: "{{ var('zendesk_schedule_holiday_identifier', 'schedule_holiday') }}"
         description: Information about holidays for each specified schedule.
         freshness: null
-        meta:
-          is_enabled: "{{ var('using_schedules', true) }}"
+        config:
+          enabled: "{{ var('using_schedules', true) }}"
         columns:
           - name: end_date
             description: ISO 8601 representation of the holiday end date.

models/stg_zendesk__user.sql

@@ -39,11 +39,18 @@ final as (
             last_login_at,
             created_at,
             updated_at,
-        {% endif %}
+        {% endif -%}
         email,
         name,
         organization_id,
+        {% if var('internal_user_criteria', false) -%}
+            case 
+                when role in ('admin', 'agent') then role
+                when {{ var('internal_user_criteria', false) }} then 'agent'
+            else role end as role,
+        {% else -%}
         role,
+        {% endif -%}
         ticket_restriction,
         time_zone,
         locale,