Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/historical schedules #171

Merged
merged 82 commits into from
Oct 10, 2024
Merged
Changes from 2 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
605a7fb
send to bk
fivetran-jamie Jun 3, 2024
c58b4b6
try this out
fivetran-jamie Jun 3, 2024
5f2113b
first try
fivetran-jamie Jun 3, 2024
662f3f2
use mcro
fivetran-jamie Jun 3, 2024
5bfead9
Testing
fivetran-jamie Jun 3, 2024
d36dc49
puhs
fivetran-jamie Jun 3, 2024
700589f
redshift
fivetran-jamie Jun 3, 2024
f6ada0c
try nullif
fivetran-jamie Jun 4, 2024
f07271c
feature/historical-schedules
fivetran-catfritz Sep 11, 2024
f15b4b1
Merge branch 'feature/historical-schedules' into explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
fd53978
Merge pull request #170 from fivetran/explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
cf89429
rework schedule_history
fivetran-catfritz Sep 12, 2024
54cdf21
update unnest logic
fivetran-catfritz Sep 13, 2024
505f343
update schedule_history
fivetran-catfritz Sep 23, 2024
4ca4099
complete schedule_history
fivetran-catfritz Sep 23, 2024
7806d83
revise holidays
fivetran-catfritz Sep 26, 2024
a54bee2
add macro
fivetran-catfritz Sep 26, 2024
0477543
typeo
fivetran-catfritz Sep 26, 2024
f980433
updates
fivetran-catfritz Sep 26, 2024
016e449
updates
fivetran-catfritz Sep 26, 2024
45fbcfe
updates
fivetran-catfritz Sep 26, 2024
3907813
updates
fivetran-catfritz Sep 26, 2024
8b32c45
try out in buildkite
fivetran-jamie Sep 27, 2024
9c49399
remove schedule days during holiday
fivetran-catfritz Sep 27, 2024
e4bf09f
revise
fivetran-catfritz Sep 27, 2024
4523d2c
revise
fivetran-catfritz Sep 27, 2024
cceae3f
add config
fivetran-catfritz Sep 27, 2024
62c5fa9
add config
fivetran-catfritz Sep 27, 2024
1db42a7
allow disable holidays
fivetran-catfritz Sep 27, 2024
f730908
streamline bk run
fivetran-jamie Sep 27, 2024
ee99616
adjust for multiple holidays in a week
fivetran-catfritz Sep 27, 2024
4f395de
add casting
fivetran-catfritz Sep 27, 2024
8613f8a
add casting
fivetran-catfritz Sep 27, 2024
f762ca1
fixes
fivetran-catfritz Sep 27, 2024
00a98cb
fixes
fivetran-catfritz Sep 27, 2024
96be5ae
fixes
fivetran-catfritz Sep 27, 2024
7517057
fixes
fivetran-catfritz Sep 27, 2024
184e639
fixes
fivetran-catfritz Sep 28, 2024
c594f3f
fix multiyear schedules
fivetran-catfritz Sep 28, 2024
e613480
add longer holiday support
fivetran-catfritz Sep 30, 2024
a49f0d6
revert multiweek
fivetran-catfritz Oct 1, 2024
e1f9d32
adjust multiweek
fivetran-catfritz Oct 1, 2024
98e6872
account for non sunday week starts
fivetran-catfritz Oct 2, 2024
0321581
updates
fivetran-catfritz Oct 2, 2024
6163942
update weeks spanned calc
fivetran-catfritz Oct 2, 2024
c774f6d
update naming
fivetran-catfritz Oct 2, 2024
b8a9f4a
update to dbt_date
fivetran-catfritz Oct 3, 2024
cfcd106
update comments
fivetran-catfritz Oct 3, 2024
781f4fd
update to dbt date weekstart
fivetran-catfritz Oct 3, 2024
28457e8
modernize calendar spine
fivetran-catfritz Oct 3, 2024
c117df9
make sure we're working with strings when replacing
fivetran-jamie Oct 3, 2024
933d62e
let's see if bk works
fivetran-jamie Oct 4, 2024
0aacff4
postgres?
fivetran-jamie Oct 4, 2024
e39919e
i think schedule history may actually be workingggg
fivetran-catfritz Oct 4, 2024
8a9d84d
get ready to merge into catherines branch
fivetran-jamie Oct 4, 2024
f691904
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-jamie Oct 4, 2024
b3c97ab
postgres revert
fivetran-jamie Oct 4, 2024
94b11a2
Merge branch 'feature/historical-schedules-jamie-redshift' of https:/…
fivetran-jamie Oct 4, 2024
eff6401
add comments
fivetran-catfritz Oct 4, 2024
763ca33
redshift fixes
fivetran-catfritz Oct 4, 2024
238bd96
redshift fixes
fivetran-catfritz Oct 4, 2024
4f57203
redshift fixes
fivetran-catfritz Oct 4, 2024
a6f3536
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-catfritz Oct 4, 2024
e342c79
Merge pull request #172 from fivetran/feature/historical-schedules-ja…
fivetran-catfritz Oct 4, 2024
5c720b5
updates
fivetran-catfritz Oct 4, 2024
6d8ff79
validation update
fivetran-catfritz Oct 6, 2024
c100e51
split models
fivetran-catfritz Oct 6, 2024
56bb954
update ymls
fivetran-catfritz Oct 7, 2024
c4929cf
revise scchedule groups
fivetran-catfritz Oct 8, 2024
1d1b2e1
updates
fivetran-catfritz Oct 9, 2024
05fefd5
add inline comments
fivetran-catfritz Oct 9, 2024
d97f714
update decision log
fivetran-catfritz Oct 9, 2024
6573314
update changelog
fivetran-catfritz Oct 9, 2024
0b0a05d
regen docs
fivetran-catfritz Oct 9, 2024
246624b
fix yml
fivetran-catfritz Oct 9, 2024
888f109
update changelog
fivetran-catfritz Oct 9, 2024
af951b1
address review comments and regen docs
fivetran-catfritz Oct 9, 2024
855dc77
address review comments
fivetran-catfritz Oct 9, 2024
73845a9
Apply suggestions from code review
fivetran-catfritz Oct 10, 2024
fd879c5
address review comments
fivetran-catfritz Oct 10, 2024
360911f
Update packages.yml
fivetran-catfritz Oct 10, 2024
56cd177
release review updates
fivetran-catfritz Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions models/intermediate/int_zendesk__timezones_w_dt.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
{{ config(enabled=var('using_schedules', True)) }}

/*
The purpose of this model is to create a spine of appropriate timezone offsets to use for schedules, as offsets may change due to Daylight Savings.
End result will include `valid_from` and `valid_until` columns which we will use downstream to determine which schedule-offset to associate with each ticket (ie standard time vs daylight time)
*/

with timezone as (

select *
from {{ var('time_zone') }}

), daylight_time as (

select *
from {{ var('daylight_time') }}

), timezone_with_dt as (

select
timezone.*,
daylight_time.daylight_start_utc,
daylight_time.daylight_end_utc,
daylight_time.daylight_offset_minutes

from timezone
left join daylight_time
on timezone.time_zone = daylight_time.time_zone

), order_timezone_dt as (

select
*,
-- will be null for timezones without any daylight savings records (and the first entry)
-- we will coalesce the first entry date with .... the X years ago
lag(daylight_end_utc, 1) over (partition by time_zone order by daylight_end_utc asc) as last_daylight_end_utc,
-- will be null for timezones without any daylight savings records (and the last entry)
-- we will coalesce the last entry date with the current date
lead(daylight_start_utc, 1) over (partition by time_zone order by daylight_start_utc asc) as next_daylight_start_utc

from timezone_with_dt

), split_timezones as (

-- standard (includes timezones without DT)
-- starts: when the last Daylight Savings ended
-- ends: when the next Daylight Savings starts
select
time_zone,
standard_offset_minutes as offset_minutes,

-- last_daylight_end_utc is null for the first record of the time_zone's daylight time, or if the TZ doesn't use DT
coalesce(last_daylight_end_utc, cast('1970-01-01' as date)) as valid_from,

-- daylight_start_utc is null for timezones that don't use DT
coalesce(daylight_start_utc, cast( {{ dbt.dateadd('year', 1, dbt.current_timestamp_backcompat()) }} as date)) as valid_until

from order_timezone_dt

union all

-- DT (excludes timezones without it)
-- starts: when this Daylight Savings started
-- ends: when this Daylight Savings ends
select
time_zone,
-- Pacific Time is -8h during standard time and -7h during DT
standard_offset_minutes + daylight_offset_minutes as offset_minutes,
daylight_start_utc as valid_from,
daylight_end_utc as valid_until

from order_timezone_dt
where daylight_offset_minutes is not null

union all

select
time_zone,
standard_offset_minutes as offset_minutes,

-- Get the latest daylight_end_utc time and set that as the valid_from
max(daylight_end_utc) as valid_from,

-- If the latest_daylight_end_time_utc is less than todays timestamp, that means DST has ended. Therefore, we will make the valid_until in the future.
cast( {{ dbt.dateadd('year', 1, dbt.current_timestamp_backcompat()) }} as date) as valid_until

from order_timezone_dt
group by 1, 2
-- We only want to apply this logic to time_zone's that had daylight saving time and it ended at a point. For example, Hong Kong ended DST in 1979.
having cast(max(daylight_end_utc) as date) < cast({{ dbt.current_timestamp_backcompat() }} as date)
)

select *
from split_timezones
3 changes: 1 addition & 2 deletions packages.yml
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
packages:
# - package: fivetran/zendesk_source
# version: [">=0.11.0", "<0.12.0"]
# version: [">=0.12.0", "<0.13.0"]
- git: https://github.com/fivetran/dbt_zendesk_source.git
revision: explore/audit-log-spike
warn-unpinned: false
# - local: ../dbt_zendesk_source
- package: calogica/dbt_date
version: [">=0.9.0", "<1.0.0"]