Jmcneill/fix get column values #533
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
5 tasks
jpmmcneill
commented
Mar 31, 2022
| {#-- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. #} | ||
| {%- if not execute -%} | ||
| {% set default = [] if not default %} |
There was a problem hiding this comment.
return(default or []) is definitely less code, but I personally find this slightly more readable. Happy to change if lower code is important to you @joellabes 😄
There was a problem hiding this comment.
I don't have strong feelings either way! This is all good 👌
joellabes
approved these changes
Apr 6, 2022
There was a problem hiding this comment.
Thank you @jpmmcneill! This is a great fix 🎉
On the issue, you asked
Do you think we should swap out load_relation to get_or_create_relation?
I had a look into it, and I'm almost certain that get_or_create_relation would mean that relation_exists would always be true, so we wouldn't hit any of the safety checks anymore.
get_or_create_relation will create a Relation object if it doesn't exist (code), which means that the is not none check will never be true. Am I interpreting that correctly?
I'm going to merge this in as-is to fix the existing bug, and if there's something I've gotten wrong then we can look at swapping out the behaviour in a future PR. Thanks again!
| {#-- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. #} | ||
| {%- if not execute -%} | ||
| {% set default = [] if not default %} |
There was a problem hiding this comment.
I don't have strong feelings either way! This is all good 👌
|
Cheers @joellabes. Happy to help! Re. The create relation - it seemed that the problem that I ran into was caused by a cache not being updated (basically, a fresh context of DBT where a given model was not being run, but being queried as a relation). I think that that swap would help in that use case, but it's probably a bit of an overcorrection. I think it could have been gotten around via a --bypass-cache flag in the given run |
dbeatty10
pushed a commit
that referenced
this pull request
Apr 7, 2022
* Fix/timestamp withought timezone (#458) * timestamp and changelog updates * changelog fix * Add context for why change to no timezone Co-authored-by: Joel Labes <[email protected]> * also ignore dbt_packages (#463) * also ignore dbt_packages * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joel Labes <[email protected]> * date_spine: transform comment to jinja (#462) * Have union_relations raise exception when include parameter results in no columns (#473) * Raise exception if no columns in column_superset * Add relation names to compiler error message * Add `union_relations` fix to changelog * Added case for handling postgres foreign tables... (#476) * Add link for fewer_rows_than schema test in docs (#465) * Added case for handling postgres foreign tables (tables which are external to current database and are imported into current database from remote data stores by using Foreign Data Wrappers functionallity). * Reworked getting of postges table_type. * Added needed changes to CHANGELOG. Co-authored-by: José Coto <[email protected]> Co-authored-by: Taras Stetsiak <[email protected]> * Enhance usability of star macro by only generating column aliases when prefix and/or suffix is specified (#468) * The star macro should only produce column aliases when there is either a prefix or suffix specified. * Enhanced the readme for the star macro. * Add new integration test Co-authored-by: Nick Perrott <[email protected]> Co-authored-by: Josh Elston-Green Co-authored-by: Joel Labes <[email protected]> * fix: extra brace typo in insert_by_period_materialization (#480) * Support quoted column names in sequential_values test (#479) * Add any value (#501) * Add link for fewer_rows_than schema test in docs (#465) * Update get_query_results_as_dict example to demonstrate accessing columnar results as dictionary values (#474) * Update get_qu ery_results_as_dict example to demonstrate accessing columnar results as dictionary values * Use slugify in example * Fix slugify example with dbt_utils. package prefix Co-authored-by: Elize Papineau <[email protected]> * Add note about not_null_where deprecation to Readme (#477) * Add note about not_null_where deprecation to Readme * Add docs to unique_where test * Update pull_request_template.md to reference `main` vs `master` (#496) * Correct coalesce -> concatenation typo (#495) * add any_value cross-db macro * Missing colon in test * Update CHANGELOG.md Co-authored-by: José Coto <[email protected]> Co-authored-by: Elize Papineau <[email protected]> Co-authored-by: Elize Papineau <[email protected]> Co-authored-by: Joe Ste.Marie <[email protected]> Co-authored-by: Niall Woodward <[email protected]> * Fix changelog * Second take at fixing pivot to allow single quotes (#503) * fix pivot : in pivoted column value, single quote must be escaped (on postgresql) else ex. syntax error near : when color = 'blue's' * patched expected * single quote escape : added dispatched version of the macro to support bigquery & snowflake * second backslash to escape in Jinja, change case of test file columns Let's see if other databases allow this * explicitly list columns to compare * different tests for snowflake and others * specific comparison seed * Don't quote identifiers for apostrophe, to avoid BQ and SF problems * Whitespace management for macros * Update CHANGELOG.md Co-authored-by: Marc Dutoo <[email protected]> * Add bool or cross db (#504) * Create bool_or cross-db func * Forgot a comma * Update CHANGELOG.md * Code review tweaks * Fix union_relations error when no include/exclude provided (#509) * Update CHANGELOG.md * Add _is_ephemeral test to get_column_values (#518) * Add _is_ephemeral test Co-authored-by: Elize Papineau <[email protected]> * Add deduplication macro (#512) * Update README.md * Mutually excl range examples in disclosure triangle * Fix union_relations error when no include/exclude provided * Fix union_relations error when no include/exclude provided (#509) * Update CHANGELOG.md * Add dedupe macro * Add test for dedupe macro * Add documentation to README * Add entry to CHANGELOG * Implement review * Typed materialized views as views (#525) * Typed materialized views as views * Update get_relations_by_pattern.sql * Moving fix from get_tables_by_pattern_sql reverting changes to this file to add a fix to the macro get_tables_by_pattern_sql * removing quoting from table_type removing quoting from table_type as this was causing an error when calling this macro within get_tables_by_pattern_sql * calling get_table_types_sql for materialized views calling get_table_types_sql macro to handle materialized views in sources. * Add `alias` argument to `deduplicate` macro (#526) * Add `alias` argument to `deduplicate * Test `alias` argument * Rename `alias` to `relation_alias` * Fix/use generic test naming style instead of schema test (#521) * Updated Rreferences to 'schema test' in README along with small improvements to test descriptions. Updates were also carried out in folder structure and integration README * Updated references to 'schema test' in Changelog * updated changelog with changes to documentation and fproject file structure * Apply suggestions from code review Update macro descriptions to be "asserts that" * Update CHANGELOG.md * Update README.md Co-authored-by: Joel Labes <[email protected]> * Remove extraneous whitespace (#529) * rm whitespace from date_trunc * datediff * rm uncessary whitespace control * change log * fix CHANGELOG * address comments * Feature/add listagg macro (#530) * Update README.md * Mutually excl range examples in disclosure triangle * Fix union_relations error when no include/exclude provided * Fix union_relations error when no include/exclude provided (#509) * Update CHANGELOG.md * Add to_condition to relationships where * very minor nit - update "an new" to "a new" (#519) * add quoting to split_part (#528) * add quoting to split_part * update docs for split_part * typo * corrected readme syntax * revert and update to just documentation * add new line * Update README.md * Update README.md * Update README.md Co-authored-by: Joel Labes <[email protected]> * add macro to get columns (#516) * add macro to get columns * star macro should use get_columns * add adapter. * swap adapter for dbt_utils Co-authored-by: Joel Labes <[email protected]> * update documentation * add output_lower arg * update name to get_filtered_columns_in_relation from get_columns * add tests * forgot args * too much whitespace removal ----------- Actual: ----------- --->"field_3"as "test_field_3"<--- ----------- Expected: ----------- --->"field_3" as "test_field_3"<--- * didnt mean to move a file that i did not create. moving things back. * remove lowercase logic * limit_zero Co-authored-by: Joel Labes <[email protected]> * Add listagg macro and integration test * remove type in listagg macro * updated integration test * Add redshift to listagg macro * remove redshift listagg * explicitly named group by column * updated default values * Updated example to use correct double vs. single quotes * whitespace control * Added redshift specific macro * Remove documentation * Update integration test so less likely to accidentally work Co-authored-by: Joel Labes <[email protected]> * default everything but measure to none * added limit functionality for other dbs * syntax bug for postgres * update redshift macro * fixed block def control * Fixed bug in redshift * Bug fix redshift * remove unused group_by arg * Added additional test without order by col * updated to regex replace * typo * added more integration_tests * attempt to make redshift less complicated * typo * update redshift * replace to substr * More explicit versions with added complexity * handle special characters Co-authored-by: Joel Labes <[email protected]> Co-authored-by: Jamie Rosenberg <[email protected]> Co-authored-by: Pat Kearns <[email protected]> * patch default behaviour in get_column_values (#533) * Update changelog, add missing quotes around get_table_types_sql * rm whitespace Co-authored-by: Joe Markiewicz <[email protected]> Co-authored-by: Anders <[email protected]> Co-authored-by: Mikaël Simarik <[email protected]> Co-authored-by: Graham Wetzler <[email protected]> Co-authored-by: Taras <[email protected]> Co-authored-by: José Coto <[email protected]> Co-authored-by: Taras Stetsiak <[email protected]> Co-authored-by: nickperrott <[email protected]> Co-authored-by: Nick Perrott <[email protected]> Co-authored-by: Ted Conbeer <[email protected]> Co-authored-by: Armand Duijn <[email protected]> Co-authored-by: Elize Papineau <[email protected]> Co-authored-by: Elize Papineau <[email protected]> Co-authored-by: Joe Ste.Marie <[email protected]> Co-authored-by: Niall Woodward <[email protected]> Co-authored-by: Marc Dutoo <[email protected]> Co-authored-by: Judah Rand <[email protected]> Co-authored-by: Luis Leon <[email protected]> Co-authored-by: Brid Moynihan <[email protected]> Co-authored-by: SunriseLong <[email protected]> Co-authored-by: Grace Goheen <[email protected]> Co-authored-by: Jamie Rosenberg <[email protected]> Co-authored-by: Pat Kearns <[email protected]> Co-authored-by: James McNeill <[email protected]>
5 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a:
mainDescription & motivation
Bugfix for: #531
Checklist
star()source)limit_zero()macro in place of the literal string:limit 0dbt_utils.type_*macros instead of explicit datatypes (e.g.dbt_utils.type_timestamp()instead ofTIMESTAMP