-
Notifications
You must be signed in to change notification settings - Fork 29
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Loading status checks…
Release/v0.17.0 (#169)
* initial * feature/unstructured-data * add coalesce_cast * update filters * update and consolidate models * model revisions * restructure * documentation * remove extra comma * regen docs * formatting * update max token docs * Update CHANGELOG.md * bug/missing-sla-policies * update changelog and add integrity test * update test * update changelog, readme and tests * update test * bug/intercepted-period-joins * adjustmnt * update weeks * update weeks * add integrity test * update weeks * update changelog * bugfix/too-many-partitions (#165) * bugfix/too-many-partitions * docs regen * Update CHANGELOG.md Co-authored-by: fivetran-catfritz <[email protected]> --------- Co-authored-by: fivetran-catfritz <[email protected]> * update changelog * revert docs to main * Documentation Standard Updates (#166) * MagicBot/documentation-updates * Apply suggestions from code review * Update README.md Co-authored-by: fivetran-catfritz <[email protected]> --------- Co-authored-by: fivetran-catfritz <[email protected]> * update default max_tokens * update changelog * Apply suggestions from code review Co-authored-by: Joe Markiewicz <[email protected]> * update readme * regen docs * update yml * Apply suggestions from code review Co-authored-by: Renee Li <[email protected]> * add comments and update changelog * update changelog * Update packages.yml --------- Co-authored-by: Renee Li <[email protected]> Co-authored-by: Joe Markiewicz <[email protected]> Co-authored-by: Renee Li <[email protected]>
1 parent
66d7b7c
commit 0b73c19
Showing
40 changed files
with
540 additions
and
157 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
|
||
{{ config( | ||
tags="fivetran_validations", | ||
enabled=var('fivetran_validation_tests_enabled', false) | ||
) }} | ||
|
||
-- check that all the tickets are accounted for in the metrics | ||
with stg_count as ( | ||
select | ||
count(*) as stg_ticket_count | ||
from {{ ref('stg_zendesk__ticket') }} | ||
), | ||
|
||
metric_count as ( | ||
select | ||
count(*) as metric_ticket_count | ||
from source | ||
from {{ ref('zendesk__ticket_metrics') }} | ||
) | ||
|
||
select * | ||
from stg_count | ||
join metric_count | ||
on stg_ticket_count != metric_ticket_count |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
{{ config( | ||
tags="fivetran_validations", | ||
enabled=var('fivetran_validation_tests_enabled', false) | ||
) }} | ||
|
||
/* | ||
This test is to ensure the sla_elapsed_time from zendesk__sla_policies matches the corresponding time in zendesk__ticket_metrics. | ||
*/ | ||
|
||
with dev_slas as ( | ||
select * | ||
from {{ target.schema }}_zendesk_dev.zendesk__sla_policies | ||
where in_business_hours | ||
|
||
), dev_metrics as ( | ||
select * | ||
from {{ target.schema }}_zendesk_dev.zendesk__ticket_metrics | ||
|
||
), dev_compare as ( | ||
select | ||
dev_slas.ticket_id, | ||
dev_slas.metric, | ||
cast(dev_slas.sla_elapsed_time as {{ dbt.type_int() }}) as time_from_slas, | ||
case when metric = 'agent_work_time' then dev_metrics.agent_work_time_in_business_minutes | ||
when metric = 'requester_wait_time' then dev_metrics.requester_wait_time_in_business_minutes | ||
when metric = 'first_reply_time' then dev_metrics.first_reply_time_business_minutes | ||
end as time_from_metrics | ||
from dev_slas | ||
left join dev_metrics | ||
on dev_metrics.ticket_id = dev_slas.ticket_id | ||
) | ||
|
||
select * | ||
from dev_compare | ||
where abs(time_from_slas - time_from_metrics) >= 5 | ||
{{ "and ticket_id not in " ~ var('fivetran_integrity_sla_metric_parity_exclusion_tickets',[]) ~ "" if var('fivetran_integrity_sla_metric_parity_exclusion_tickets',[]) }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
{% macro coalesce_cast(column_list, datatype) -%} | ||
{{ return(adapter.dispatch('coalesce_cast', 'zendesk')(column_list, datatype)) }} | ||
{%- endmacro %} | ||
|
||
{% macro default__coalesce_cast(column_list, datatype) %} | ||
coalesce( | ||
{%- for column in column_list %} | ||
cast({{ column }} as {{ datatype }}) | ||
{%- if not loop.last -%},{%- endif -%} | ||
{% endfor %} | ||
) | ||
{% endmacro %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
{% macro count_tokens(column_name) -%} | ||
{{ return(adapter.dispatch('count_tokens', 'zendesk')(column_name)) }} | ||
{%- endmacro %} | ||
|
||
{% macro default__count_tokens(column_name) %} | ||
{{ dbt.length(column_name) }} / 4 -- 1 token is approximately 4 characters, and we only need an approximation here. | ||
{% endmacro %} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
60 changes: 60 additions & 0 deletions
60
models/unstructured/intermediate/int_zendesk__ticket_comment_document.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,60 @@ | ||
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }} | ||
|
||
with ticket_comments as ( | ||
select * | ||
from {{ var('ticket_comment') }} | ||
|
||
), users as ( | ||
select * | ||
from {{ var('user') }} | ||
|
||
), comment_details as ( | ||
select | ||
ticket_comments.ticket_comment_id, | ||
ticket_comments.ticket_id, | ||
{{ zendesk.coalesce_cast(["users.email", "'UNKNOWN'"], dbt.type_string()) }} as commenter_email, | ||
{{ zendesk.coalesce_cast(["users.name", "'UNKNOWN'"], dbt.type_string()) }} as commenter_name, | ||
ticket_comments.created_at as comment_time, | ||
ticket_comments.body as comment_body | ||
from ticket_comments | ||
left join users | ||
on ticket_comments.user_id = users.user_id | ||
where not coalesce(ticket_comments._fivetran_deleted, False) | ||
and not coalesce(users._fivetran_deleted, False) | ||
|
||
), comment_markdowns as ( | ||
select | ||
ticket_comment_id, | ||
ticket_id, | ||
comment_time, | ||
cast( | ||
{{ dbt.concat([ | ||
"'### message from '", "commenter_name", "' ('", "commenter_email", "')\\n'", | ||
"'##### sent @ '", "comment_time", "'\\n'", | ||
"comment_body" | ||
]) }} as {{ dbt.type_string() }}) | ||
as comment_markdown | ||
from comment_details | ||
|
||
), comments_tokens as ( | ||
select | ||
*, | ||
{{ zendesk.count_tokens("comment_markdown") }} as comment_tokens | ||
from comment_markdowns | ||
|
||
), truncated_comments as ( | ||
select | ||
ticket_comment_id, | ||
ticket_id, | ||
comment_time, | ||
case when comment_tokens > {{ var('zendesk_max_tokens', 5000) }} then left(comment_markdown, {{ var('zendesk_max_tokens', 5000) }} * 4) -- approximate 4 characters per token | ||
else comment_markdown | ||
end as comment_markdown, | ||
case when comment_tokens > {{ var('zendesk_max_tokens', 5000) }} then {{ var('zendesk_max_tokens', 5000) }} | ||
else comment_tokens | ||
end as comment_tokens | ||
from comments_tokens | ||
) | ||
|
||
select * | ||
from truncated_comments |
32 changes: 32 additions & 0 deletions
32
models/unstructured/intermediate/int_zendesk__ticket_comment_documents_grouped.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }} | ||
|
||
with filtered_comment_documents as ( | ||
select * | ||
from {{ ref('int_zendesk__ticket_comment_document') }} | ||
), | ||
|
||
grouped_comment_documents as ( | ||
select | ||
ticket_id, | ||
comment_markdown, | ||
comment_tokens, | ||
comment_time, | ||
sum(comment_tokens) over ( | ||
partition by ticket_id | ||
order by comment_time | ||
rows between unbounded preceding and current row | ||
) as cumulative_length | ||
from filtered_comment_documents | ||
) | ||
|
||
select | ||
ticket_id, | ||
cast({{ dbt_utils.safe_divide('floor(cumulative_length - 1)', var('zendesk_max_tokens', 5000)) }} as {{ dbt.type_int() }}) as chunk_index, | ||
{{ dbt.listagg( | ||
measure="comment_markdown", | ||
delimiter_text="'\\n\\n---\\n\\n'", | ||
order_by_clause="order by comment_time" | ||
) }} as comments_group_markdown, | ||
sum(comment_tokens) as chunk_tokens | ||
from grouped_comment_documents | ||
group by 1,2 |
42 changes: 42 additions & 0 deletions
42
models/unstructured/intermediate/int_zendesk__ticket_document.sql
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }} | ||
|
||
with tickets as ( | ||
select * | ||
from {{ var('ticket') }} | ||
|
||
), users as ( | ||
select * | ||
from {{ var('user') }} | ||
|
||
), ticket_details as ( | ||
select | ||
tickets.ticket_id, | ||
tickets.subject AS ticket_name, | ||
{{ zendesk.coalesce_cast(["users.name", "'UNKNOWN'"], dbt.type_string()) }} as user_name, | ||
{{ zendesk.coalesce_cast(["users.email", "'UNKNOWN'"], dbt.type_string()) }} as created_by, | ||
tickets.created_at AS created_on, | ||
{{ zendesk.coalesce_cast(["tickets.status", "'UNKNOWN'"], dbt.type_string()) }} as status, | ||
{{ zendesk.coalesce_cast(["tickets.priority", "'UNKNOWN'"], dbt.type_string()) }} as priority | ||
from tickets | ||
left join users | ||
on tickets.requester_id = users.user_id | ||
where not coalesce(tickets._fivetran_deleted, False) | ||
and not coalesce(users._fivetran_deleted, False) | ||
|
||
), final as ( | ||
select | ||
ticket_id, | ||
{{ dbt.concat([ | ||
"'# Ticket : '", "ticket_name", "'\\n\\n'", | ||
"'Created By : '", "user_name", "' ('", "created_by", "')\\n'", | ||
"'Created On : '", "created_on", "'\\n'", | ||
"'Status : '", "status", "'\\n'", | ||
"'Priority : '", "priority" | ||
]) }} as ticket_markdown | ||
from ticket_details | ||
) | ||
|
||
select | ||
*, | ||
{{ zendesk.count_tokens("ticket_markdown") }} as ticket_tokens | ||
from final |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
{{ config(enabled=var('zendesk__unstructured_enabled', False)) }} | ||
|
||
with ticket_document as ( | ||
select * | ||
from {{ ref('int_zendesk__ticket_document') }} | ||
|
||
), grouped as ( | ||
select * | ||
from {{ ref('int_zendesk__ticket_comment_documents_grouped') }} | ||
|
||
), final as ( | ||
select | ||
cast(ticket_document.ticket_id as {{ dbt.type_string() }}) as document_id, | ||
grouped.chunk_index, | ||
grouped.chunk_tokens as chunk_tokens_approximate, | ||
{{ dbt.concat([ | ||
"ticket_document.ticket_markdown", | ||
"'\\n\\n## COMMENTS\\n\\n'", | ||
"grouped.comments_group_markdown"]) }} | ||
as chunk | ||
from ticket_document | ||
join grouped | ||
on grouped.ticket_id = ticket_document.ticket_id | ||
) | ||
|
||
select * | ||
from final |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,14 @@ | ||
version: 2 | ||
|
||
models: | ||
- name: zendesk__document | ||
description: Each record represents a Zendesk ticket, enriched with data about it's tags, assignees, requester, submitter, organization and group. | ||
columns: | ||
- name: document_id | ||
description: Equivalent to `ticket_id`. | ||
- name: chunk_index | ||
description: The index of the chunk associated with the `document_id`. | ||
- name: chunk_tokens_approximate | ||
description: Approximate number of tokens for the chunk, assuming 4 characters per token. | ||
- name: chunk | ||
description: The text of the chunk. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,5 @@ | ||
packages: | ||
- package: fivetran/zendesk_source | ||
version: [">=0.11.0", "<0.12.0"] | ||
|
||
version: [">=0.12.0", "<0.13.0"] | ||
- package: calogica/dbt_date | ||
version: [">=0.9.0", "<1.0.0"] |