Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/historical schedules #171

Merged
merged 82 commits into from
Oct 10, 2024
Merged
Changes from 1 commit
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
605a7fb
send to bk
fivetran-jamie Jun 3, 2024
c58b4b6
try this out
fivetran-jamie Jun 3, 2024
5f2113b
first try
fivetran-jamie Jun 3, 2024
662f3f2
use mcro
fivetran-jamie Jun 3, 2024
5bfead9
Testing
fivetran-jamie Jun 3, 2024
d36dc49
puhs
fivetran-jamie Jun 3, 2024
700589f
redshift
fivetran-jamie Jun 3, 2024
f6ada0c
try nullif
fivetran-jamie Jun 4, 2024
f07271c
feature/historical-schedules
fivetran-catfritz Sep 11, 2024
f15b4b1
Merge branch 'feature/historical-schedules' into explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
fd53978
Merge pull request #170 from fivetran/explore/audit-log-spike
fivetran-catfritz Sep 11, 2024
cf89429
rework schedule_history
fivetran-catfritz Sep 12, 2024
54cdf21
update unnest logic
fivetran-catfritz Sep 13, 2024
505f343
update schedule_history
fivetran-catfritz Sep 23, 2024
4ca4099
complete schedule_history
fivetran-catfritz Sep 23, 2024
7806d83
revise holidays
fivetran-catfritz Sep 26, 2024
a54bee2
add macro
fivetran-catfritz Sep 26, 2024
0477543
typeo
fivetran-catfritz Sep 26, 2024
f980433
updates
fivetran-catfritz Sep 26, 2024
016e449
updates
fivetran-catfritz Sep 26, 2024
45fbcfe
updates
fivetran-catfritz Sep 26, 2024
3907813
updates
fivetran-catfritz Sep 26, 2024
8b32c45
try out in buildkite
fivetran-jamie Sep 27, 2024
9c49399
remove schedule days during holiday
fivetran-catfritz Sep 27, 2024
e4bf09f
revise
fivetran-catfritz Sep 27, 2024
4523d2c
revise
fivetran-catfritz Sep 27, 2024
cceae3f
add config
fivetran-catfritz Sep 27, 2024
62c5fa9
add config
fivetran-catfritz Sep 27, 2024
1db42a7
allow disable holidays
fivetran-catfritz Sep 27, 2024
f730908
streamline bk run
fivetran-jamie Sep 27, 2024
ee99616
adjust for multiple holidays in a week
fivetran-catfritz Sep 27, 2024
4f395de
add casting
fivetran-catfritz Sep 27, 2024
8613f8a
add casting
fivetran-catfritz Sep 27, 2024
f762ca1
fixes
fivetran-catfritz Sep 27, 2024
00a98cb
fixes
fivetran-catfritz Sep 27, 2024
96be5ae
fixes
fivetran-catfritz Sep 27, 2024
7517057
fixes
fivetran-catfritz Sep 27, 2024
184e639
fixes
fivetran-catfritz Sep 28, 2024
c594f3f
fix multiyear schedules
fivetran-catfritz Sep 28, 2024
e613480
add longer holiday support
fivetran-catfritz Sep 30, 2024
a49f0d6
revert multiweek
fivetran-catfritz Oct 1, 2024
e1f9d32
adjust multiweek
fivetran-catfritz Oct 1, 2024
98e6872
account for non sunday week starts
fivetran-catfritz Oct 2, 2024
0321581
updates
fivetran-catfritz Oct 2, 2024
6163942
update weeks spanned calc
fivetran-catfritz Oct 2, 2024
c774f6d
update naming
fivetran-catfritz Oct 2, 2024
b8a9f4a
update to dbt_date
fivetran-catfritz Oct 3, 2024
cfcd106
update comments
fivetran-catfritz Oct 3, 2024
781f4fd
update to dbt date weekstart
fivetran-catfritz Oct 3, 2024
28457e8
modernize calendar spine
fivetran-catfritz Oct 3, 2024
c117df9
make sure we're working with strings when replacing
fivetran-jamie Oct 3, 2024
933d62e
let's see if bk works
fivetran-jamie Oct 4, 2024
0aacff4
postgres?
fivetran-jamie Oct 4, 2024
e39919e
i think schedule history may actually be workingggg
fivetran-catfritz Oct 4, 2024
8a9d84d
get ready to merge into catherines branch
fivetran-jamie Oct 4, 2024
f691904
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-jamie Oct 4, 2024
b3c97ab
postgres revert
fivetran-jamie Oct 4, 2024
94b11a2
Merge branch 'feature/historical-schedules-jamie-redshift' of https:/…
fivetran-jamie Oct 4, 2024
eff6401
add comments
fivetran-catfritz Oct 4, 2024
763ca33
redshift fixes
fivetran-catfritz Oct 4, 2024
238bd96
redshift fixes
fivetran-catfritz Oct 4, 2024
4f57203
redshift fixes
fivetran-catfritz Oct 4, 2024
a6f3536
Merge branch 'feature/historical-schedules' into feature/historical-s…
fivetran-catfritz Oct 4, 2024
e342c79
Merge pull request #172 from fivetran/feature/historical-schedules-ja…
fivetran-catfritz Oct 4, 2024
5c720b5
updates
fivetran-catfritz Oct 4, 2024
6d8ff79
validation update
fivetran-catfritz Oct 6, 2024
c100e51
split models
fivetran-catfritz Oct 6, 2024
56bb954
update ymls
fivetran-catfritz Oct 7, 2024
c4929cf
revise scchedule groups
fivetran-catfritz Oct 8, 2024
1d1b2e1
updates
fivetran-catfritz Oct 9, 2024
05fefd5
add inline comments
fivetran-catfritz Oct 9, 2024
d97f714
update decision log
fivetran-catfritz Oct 9, 2024
6573314
update changelog
fivetran-catfritz Oct 9, 2024
0b0a05d
regen docs
fivetran-catfritz Oct 9, 2024
246624b
fix yml
fivetran-catfritz Oct 9, 2024
888f109
update changelog
fivetran-catfritz Oct 9, 2024
af951b1
address review comments and regen docs
fivetran-catfritz Oct 9, 2024
855dc77
address review comments
fivetran-catfritz Oct 9, 2024
73845a9
Apply suggestions from code review
fivetran-catfritz Oct 10, 2024
fd879c5
address review comments
fivetran-catfritz Oct 10, 2024
360911f
Update packages.yml
fivetran-catfritz Oct 10, 2024
56cd177
release review updates
fivetran-catfritz Oct 10, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updates
fivetran-catfritz committed Sep 26, 2024

Verified

This commit was signed with the committer’s verified signature.
matthew-hagemann Matthew Hagemann
commit 45fbcfe79903ed0ce3e199c0d03998475ae496ef
97 changes: 39 additions & 58 deletions models/intermediate/int_zendesk__schedule_spine.sql
Original file line number Diff line number Diff line change
@@ -33,7 +33,10 @@ with schedule as (
cast(holiday.holiday_start_date_at as {{ dbt.type_timestamp() }} ) as holiday_valid_from,
cast(holiday.holiday_end_date_at as {{ dbt.type_timestamp() }}) as holiday_valid_until, -- The valid_until will then be the the day after.
cast(calendar_spine.date_day as {{ dbt.type_timestamp() }} ) as holiday_date,
{{ dbt.date_trunc("holiday.holiday_start_date_at")}}
cast({{ dbt.date_trunc("week", "holiday.holiday_start_date_at") }} as {{ dbt.type_timestamp() }}) as holiday_starting_sunday,
cast({{ dbt.dateadd("week", 1, dbt.date_trunc(
"week", "holiday.holiday_end_date_at")
) }} as {{ dbt.type_timestamp() }}) as holiday_ending_sunday,
holiday.holiday_id,
holiday.holiday_name,
holiday.schedule_id
@@ -74,8 +77,12 @@ with schedule as (
schedule_holiday.holiday_name,
schedule_holiday.holiday_valid_from,
schedule_holiday.holiday_valid_until,
schedule_holiday.holiday_starting_sunday,
schedule_holiday.holiday_ending_sunday,
calculate_schedules.valid_from as schedule_valid_from,
calculate_schedules.valid_until as schedule_valid_until
calculate_schedules.valid_until as schedule_valid_until,
cast({{ dbt.date_trunc("week", "calculate_schedules.valid_from") }} as {{ dbt.type_timestamp() }}) as schedule_starting_sunday,
cast({{ dbt.date_trunc("week", "calculate_schedules.valid_until") }} as {{ dbt.type_timestamp() }}) as schedule_ending_sunday
from calculate_schedules
left join schedule_holiday
on schedule_holiday.schedule_id = calculate_schedules.schedule_id
@@ -84,17 +91,7 @@ with schedule as (

), split_holidays as(
select
schedule_id,
time_zone,
start_time_utc,
end_time_utc,
schedule_name,
schedule_valid_from,
schedule_valid_until,
holiday_name,
holiday_date,
holiday_valid_from,
holiday_valid_until,
join_holidays.*,
case
when holiday_valid_from = holiday_date
then '0_start'
@@ -107,17 +104,7 @@ with schedule as (
union all

select
schedule_id,
time_zone,
start_time_utc,
end_time_utc,
schedule_name,
schedule_valid_from,
schedule_valid_until,
holiday_name,
holiday_date,
holiday_valid_from,
holiday_valid_until,
join_holidays.*,
case
when holiday_valid_until = holiday_date
then '1_end'
@@ -130,17 +117,7 @@ with schedule as (
union all

select
schedule_id,
time_zone,
start_time_utc,
end_time_utc,
schedule_name,
schedule_valid_from,
schedule_valid_until,
holiday_name,
holiday_date,
holiday_valid_from,
holiday_valid_until,
join_holidays.*,
cast(null as {{ dbt.type_string() }}) as holiday_start_or_end,
schedule_valid_from as valid_from,
schedule_valid_until as valid_until
@@ -149,7 +126,7 @@ with schedule as (

), valid_from_partition as(
select
*
split_holidays.*
, row_number() over (partition by schedule_id, start_time_utc, schedule_valid_from order by holiday_date, holiday_start_or_end) as valid_from_index
, count(*) over (partition by schedule_id, start_time_utc, schedule_valid_from) as max_valid_from_index
from split_holidays
@@ -164,10 +141,14 @@ with schedule as (
schedule_name,
schedule_valid_from,
schedule_valid_until,
schedule_starting_sunday,
schedule_ending_sunday,
holiday_name,
holiday_date,
holiday_valid_from,
holiday_valid_until,
holiday_starting_sunday,
holiday_ending_sunday,
case when valid_from_index = 1 and holiday_start_or_end is not null
then 'partition_start'
else holiday_start_or_end
@@ -188,10 +169,14 @@ with schedule as (
schedule_name,
schedule_valid_from,
schedule_valid_until,
schedule_starting_sunday,
schedule_ending_sunday,
holiday_name,
holiday_date,
holiday_valid_from,
holiday_valid_until,
holiday_starting_sunday,
holiday_ending_sunday,
'partition_end' as holiday_start_or_end,
valid_from,
valid_until,
@@ -212,39 +197,35 @@ with schedule as (
holiday_date,
holiday_valid_from,
holiday_valid_until,
holiday_starting_sunday,
holiday_ending_sunday,
schedule_valid_from,
schedule_valid_until,
schedule_starting_sunday,
schedule_ending_sunday,

case
when holiday_start_or_end = 'partition_start'
then cast({{ dbt.date_trunc("week", "schedule_valid_from") }} as {{ dbt.type_timestamp() }})
when holiday_start_or_end = '0_start'
then cast({{ dbt.date_trunc("week",
"lag(holiday_valid_until) over (partition by schedule_id, start_time_utc, schedule_valid_from order by valid_from_index)"
) }} as {{ dbt.type_timestamp() }})
then schedule_starting_sunday
{# when holiday_start_or_end = '0_start'
then holiday_starting_sunday #}
when holiday_start_or_end = '1_end'
then cast({{ dbt.date_trunc("week",
"lag(holiday_valid_until) over (partition by schedule_id, start_time_utc, schedule_valid_from order by valid_from_index)"
) }} as {{ dbt.type_timestamp() }})
{# then cast({{ dbt.date_trunc("week", "holiday_valid_from") }} as {{ dbt.type_timestamp() }}) #}
{# then cast(lag(valid_until) over (partition by schedule_id, start_time_utc, schedule_valid_from order by valid_from_index)
as {{ dbt.type_timestamp() }}) #}
then holiday_starting_sunday
when holiday_start_or_end = 'partition_end'
then cast({{ dbt.dateadd("week", 1, dbt.date_trunc("week", "holiday_valid_until")) }} as {{ dbt.type_timestamp() }})
else cast({{ dbt.date_trunc("week", "schedule_valid_from") }} as {{ dbt.type_timestamp() }})
then holiday_ending_sunday
else schedule_starting_sunday
end as valid_from,

case
when holiday_start_or_end = 'partition_start'
then cast({{ dbt.date_trunc("week", "holiday_valid_from") }} as {{ dbt.type_timestamp() }})
when holiday_start_or_end = '0_start'
then cast({{ dbt.dateadd("week", 1, dbt.date_trunc("week",
"lead(holiday_valid_from) over (partition by schedule_id, start_time_utc, schedule_valid_from order by valid_from_index)"
)) }} as {{ dbt.type_timestamp() }})
then holiday_starting_sunday
{# when holiday_start_or_end = '0_start'
then holiday_ending_sunday #}
when holiday_start_or_end = '1_end'
then cast({{ dbt.dateadd("week", 1, dbt.date_trunc("week", "holiday_valid_until")) }} as {{ dbt.type_timestamp() }})
{# then lead(holiday_valid_from) over (partition by schedule_id, start_time_utc, schedule_valid_from order by valid_from_index) #}
then holiday_ending_sunday
when holiday_start_or_end = 'partition_end'
then cast({{ dbt.date_trunc("week", "schedule_valid_until") }} as {{ dbt.type_timestamp() }})
else cast({{ dbt.date_trunc("week", "schedule_valid_until") }} as {{ dbt.type_timestamp() }})
then schedule_ending_sunday
else schedule_ending_sunday
end as valid_until,

valid_from_index,