Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add macro to get columns #516

Merged
merged 13 commits into from
Mar 28, 2022
27 changes: 26 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ For compatibility details between versions of dbt-core and dbt-utils, [see this

- [Introspective macros](#introspective-macros):
- [get_column_values](#get_column_values-source)
- [get_columns](#get_columns-source)
- [get_relations_by_pattern](#get_relations_by_pattern-source)
- [get_relations_by_prefix](#get_relations_by_prefix-source)
- [get_query_results_as_dict](#get_query_results_as_dict-source)
Expand Down Expand Up @@ -544,7 +545,7 @@ These macros run a query and return the results of the query as objects. They ar
#### get_column_values ([source](macros/sql/get_column_values.sql))
This macro returns the unique values for a column in a given [relation](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation) as an array.

Arguments:
**Args:**
- `table` (required): a [Relation](https://docs.getdbt.com/reference/dbt-classes#relation) (a `ref` or `source`) that contains the list of columns you wish to select from
- `column` (required): The name of the column you wish to find the column values of
- `order_by` (optional, default=`'count(*) desc'`): How the results should be ordered. The default is to order by `count(*) desc`, i.e. decreasing frequency. Setting this as `'my_column'` will sort alphabetically, while `'min(created_at)'` will sort by when thevalue was first observed.
Expand Down Expand Up @@ -585,6 +586,30 @@ Arguments:
...
```

#### get_columns ([source](macros/sql/get_columns.sql))
This macro returns the unique columns for a given [relation](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation) as a comma-separated list.
patkearns10 marked this conversation as resolved.
Show resolved Hide resolved

**Args:**
- `from` (required): a [Relation](https://docs.getdbt.com/reference/dbt-classes#relation) (a `ref` or `source`) that contains the list of columns you wish to select from
- `except` (optional, default=`[]`): The name of the columns you wish to exclude
patkearns10 marked this conversation as resolved.
Show resolved Hide resolved

{% set column_names = get_columns(from=ref('fct_rate_rapid_onboarding'), except=["rate_rapid_onboarding__survey_question_response_id", "rate_rapid_onboarding__survey_response_id"]) %}

{% for column_name in column_names %}
max({{ column_name }}) ... as '{{ column_name }}'_test,
{% endfor %}

**Usage:**
```sql
-- Returns a list of the columns from a relation, then iterate in a for loop
{% set column_names = get_columns(from=ref('your_model'), except=["field_1", "field_2"]) %}
...
{% for column_name in column_names %}
max({{ column_name }}) ... as max_'{{ column_name }}',
{% endfor %}
...
```

#### get_relations_by_pattern ([source](macros/sql/get_relations_by_pattern.sql))
Returns a list of [Relations](https://docs.getdbt.com/docs/writing-code-in-dbt/class-reference/#relation)
that match a given schema- or table-name pattern.
Expand Down
19 changes: 19 additions & 0 deletions macros/sql/get_columns.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
{% macro get_columns(from, except=[]) -%}
patkearns10 marked this conversation as resolved.
Show resolved Hide resolved

{# -- Prevent querying of db in parsing mode. This works because this macro does not create any new refs. #}
{%- if not execute -%}
{{ return('') }}
{% endif %}

{%- set include_cols = [] %}
{%- set cols = adapter.get_columns_in_relation(from) -%}
patkearns10 marked this conversation as resolved.
Show resolved Hide resolved
{%- set except = except | map("lower") | list %}
{%- for col in cols -%}
{%- if col.column|lower not in except -%}
{% do include_cols.append(col.column) %}
{%- endif %}
{%- endfor %}

{{ return(include_cols) }}

{%- endmacro %}