Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize legacy insert by period materialization #410

Closed
wants to merge 29 commits into from
Closed

Modernize legacy insert by period materialization #410

Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
3493d3f
Update legacy macro to 0.21.latest incremental
HorvathDanielMarton Aug 17, 2021
051a2e4
Raise error if no filter specified
HorvathDanielMarton Aug 17, 2021
8536054
Create empty table first
HorvathDanielMarton Aug 17, 2021
96ff9ed
Add configurations and period filter
HorvathDanielMarton Aug 23, 2021
7ac2058
Make macro work on manually defined simple example
HorvathDanielMarton Aug 23, 2021
364772b
Implement for loop with predefined time boundaries
HorvathDanielMarton Aug 23, 2021
147494a
Move period boundary macros to helpers.sql
HorvathDanielMarton Aug 23, 2021
3bb17d5
Modify helper macros to support Snowflake
HorvathDanielMarton Aug 24, 2021
a8d3259
Use macros to automate the for loop
HorvathDanielMarton Aug 24, 2021
3e100c2
Refactor and add logging, comments
HorvathDanielMarton Aug 26, 2021
760a33d
Use incremental_upsert() in the for loop
HorvathDanielMarton Aug 26, 2021
ec73c80
Introduce check_for_period_filter macro
HorvathDanielMarton Aug 26, 2021
f876d1d
Dispatch check_for_period_filter() macro
HorvathDanielMarton Aug 26, 2021
a961622
Remove unnecessary code, improve logging
HorvathDanielMarton Aug 26, 2021
ed6d55c
Handle full refresh
HorvathDanielMarton Aug 31, 2021
e53a801
Remove verbose CLI logging
HorvathDanielMarton Aug 31, 2021
b360a91
Update CHANGELOG.md
HorvathDanielMarton Aug 31, 2021
53017a4
Update README.md
HorvathDanielMarton Aug 31, 2021
888d781
Alias table expression in a helper function
HorvathDanielMarton Aug 31, 2021
6fe4adb
Update integration tests
HorvathDanielMarton Aug 31, 2021
af2dd57
Set default period to week
HorvathDanielMarton Aug 31, 2021
cdba4b5
Use make_temp_relation() in for loop
HorvathDanielMarton Aug 31, 2021
7c81020
Update README.md
HorvathDanielMarton Sep 1, 2021
f82f2ee
Run test_insert_by_period for all targets
HorvathDanielMarton Sep 1, 2021
171e5df
Merge remote-tracking branch 'upstream/master' into fix-legacy-insert…
HorvathDanielMarton Sep 1, 2021
e1cdcee
Add aggressive CLI logging for debug purposes
HorvathDanielMarton Sep 3, 2021
713f8a6
Add logging to get_period_boundaries()
HorvathDanielMarton Sep 3, 2021
1ed1e58
Extend helper function logging
HorvathDanielMarton Sep 3, 2021
9c680f9
Remove debug logging
HorvathDanielMarton Sep 3, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Set default period to week
  • Loading branch information
@HorvathDanielMarton
HorvathDanielMarton committed Aug 31, 2021
commit af2dd572cd4de375f3f2f181467e430d6776aabd
2 changes: 1 addition & 1 deletion macros/materializations/insert_by_period_materialization.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
-- Configuration (others)
{%- set unique_key = config.get('unique_key') -%}
{%- set stop_date = config.get('stop_date') or '' -%}}
{%- set period = config.get('period') or 'hour' -%}
{%- set period = config.get('period') or 'week' -%}

{% set target_relation = this.incorporate(type='table') %}
{% set existing_relation = load_relation(this) %}
Expand Down