Merge pull request #367 from fishtown-analytics/next

Release 0.18.0
dbt-labs · Sep 3, 2020 · d2b5c17 · d2b5c17
2 parents 5aebb33 + 57cb993
commit d2b5c17
Show file tree

Hide file tree

Showing 52 changed files with 1,476 additions and 322 deletions.
diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -14,7 +14,10 @@ Include any notes about things that need to happen before this PR is merged, e.g
 
 ## Pre-release docs
 Is this change related to an unreleased version of dbt?
-- [ ] Yes: please update the base branch to `next`
+- [ ] Yes: please
+    - update the base branch to `next`
+    - add Changelog components: `<Changelog>[New/Changed] in v0.x.0</Changelog>`
+    - add links to the "New and changed documentation" section of the latest [Migration Guide](../website/docs/docs/guides/migration-guide)
 - [ ] No: please ensure the base branch is `current`
 - [ ] Unsure: we'll let you know!
 

diff --git a/_redirects b/_redirects
@@ -75,7 +75,9 @@
 /docs/macros	/docs/building-a-dbt-project/macros	302
 /docs/managing-environments	/docs/guides/managing-environments	302
 /docs/materializations	/docs/building-a-dbt-project/building-models/materializations	302
-/docs/model-selection-syntax	/reference/model-selection-syntax	302
+/docs/model-selection-syntax	/reference/node-selection/syntax	302
+/reference/model-selection-syntax    /reference/node-selection/syntax 302   
+reference/model-selection-syntax/#test-selection-examples    /reference/node-selection/test-selection-examples 302 
 /docs/modules	/docs/writing-code-in-dbt/jinja-context/modules	302
 /docs/on-run-end-context	/docs/writing-code-in-dbt/jinja-context/on-run-end-context	302
 /docs/package-management	/docs/building-a-dbt-project/package-management	302

diff --git a/netlify.toml b/netlify.toml
@@ -4,4 +4,5 @@
 
 [context.next.environment]
   PRERELEASE = "true"
-  ALGOLIA_INDEX_NAME = "dbt-next"
+  ALGOLIA_API_KEY = "044414e92a8b8137a33fd276d4533895"
+  ALGOLIA_INDEX_NAME = "dbt_next"
diff --git a/website/docs/docs/building-a-dbt-project/documentation.md b/website/docs/docs/building-a-dbt-project/documentation.md
@@ -148,6 +148,32 @@ as well as the repo for this project \[here](https://github.com/fishtown-analyti
 
 </File>
 
+### Custom project-level overviews
+<Changelog>New in v0.18.0</Changelog>
+
+You can set different overviews for each dbt project/package included in your documentation site
+by creating a docs block named `__[project_name]__`. For example, in order to define
+custom overview pages that appear when a viewer navigates inside the `dbt_utils` or `snowplow` package:
+
+<File name='models/overview.md'>
+
+```markdown
+{% docs __dbt_utils__ %}
+# Utility macros
+Our dbt project heavily uses this suite of utility macros, especially:
+- `surrogate_key`
+- `test_equality`
+- `pivot`
+{% enddocs %}
+
+{% docs __snowplow__ %}
+# Snowplow sessionization
+Our organization uses this package of transformations to roll Snowplow events
+up to page views and sessions.
+{% enddocs %}
+```
+
+</File>
 
 ## Navigating the documentation site
 Using the docs interface, you can navigate to the documentation for a specific model. That might look something like this:
@@ -160,7 +186,7 @@ From a docs page, you can click the green button in the bottom-right corner of t
 
 <Lightbox src="/img/docs/building-a-dbt-project/testing-and-documentation/ec77c45-Screen_Shot_2018-08-14_at_6.31.56_PM.png" title="Opening the DAG mini-map"/>
 
-In this example, the `fct_subscription_transactions` model only has one direct parent. By clicking the "Expand" button in the top-right corner of the window, we can pivot the graph horizontally and view the full lineage for our model. This lineage is filterable using the `--models` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](model-selection-syntax). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers.
+In this example, the `fct_subscription_transactions` model only has one direct parent. By clicking the "Expand" button in the top-right corner of the window, we can pivot the graph horizontally and view the full lineage for our model. This lineage is filterable using the `--models` and `--exclude` flags, which are consistent with the semantics of [model selection syntax](node-selection/syntax). Further, you can right-click to interact with the DAG, jump to documentation, or share links to your graph visualization with your coworkers.
 
 <Lightbox src="/img/docs/building-a-dbt-project/testing-and-documentation/ac97fba-Screen_Shot_2018-08-14_at_6.35.14_PM.png" title="The full lineage for a dbt model"/>
 

diff --git a/website/docs/docs/building-a-dbt-project/tests.md b/website/docs/docs/building-a-dbt-project/tests.md
@@ -8,7 +8,7 @@ title: "Tests"
 * [Data test configurations](data-test-configs)
 
 <!---
-* [Test selection syntax](model-selection-syntax)
+* [Test selection syntax](node-selection/syntax)
 --->
 
 ## Getting started

diff --git a/website/docs/docs/dbt-cloud/cloud-ide/the-dbt-ide.md b/website/docs/docs/dbt-cloud/cloud-ide/the-dbt-ide.md
@@ -21,7 +21,7 @@ In the dbt IDE, you can compile dbt code into SQL and execute it against your da
 
 ## Running Projects
 
-In addition to compiling and executing SQL, you can also *run* dbt projects in the dbt IDE. Use dbt's [rich model selection syntax](model-selection-syntax) to [run dbt commands](dbt-commands) directly in your browser.
+In addition to compiling and executing SQL, you can also *run* dbt projects in the dbt IDE. Use dbt's [rich model selection syntax](node-selection/syntax) to [run dbt commands](dbt-commands) directly in your browser.
 
 The dbt IDE updates in real-time as models, tests, seeds, and operations are run. If a model or tests fails, you can dig into the logs to find and fix the issue.
 

diff --git a/website/docs/docs/guides/best-practices.md b/website/docs/docs/guides/best-practices.md
@@ -70,7 +70,7 @@ Complex models often include multiple Common Table Expressions (CTEs). In dbt, y
 ### Group your models in directories
 Within your `models/` directory, you can have any number of nested subdirectories. We leverage directories heavily, since using a nested structure within directories makes it easier to:
 * Configure groups of models, by specifying configurations in your `dbt_project.yml` file.
-* Run subsections of your DAG, by using the [model selection syntax](model-selection-syntax).
+* Run subsections of your DAG, by using the [model selection syntax](node-selection/syntax).
 * Communicate modeling steps to collaborators
 * Create conventions around the allowed upstream dependencies of a model, for example, "models in the `marts` directory can only select from other models in the `marts` directory, or from models in the `staging` directory".
 
@@ -104,7 +104,24 @@ We often:
 
 ## Pro-tips for workflows
 ### Use the model selection syntax when running locally
-When developing, it often makes sense to only run the model you are actively working on and any downstream models. You can choose which models to run by using the [model selection syntax](model-selection-syntax).
+When developing, it often makes sense to only run the model you are actively working on and any downstream models. You can choose which models to run by using the [model selection syntax](node-selection/syntax).
+
+### Run only modified models to test changes ("slim CI")
+To merge code changes with confidence, you want to know that those changes will not cause breakages elsewhere in your project. For that reason, we recommend running models and tests in a sandboxed environment, separated from your production data, as an automatic check in your git workflow. (If you use GitHub and dbt Cloud, read about [how to set up CI jobs](cloud-enabling-continuous-integration-with-github).)
+
+<Changelog>New in v0.18.0</Changelog>
+
+:::info [β] Beta Feature
+This is net-new functionality in v0.18.0, with iterative improvements to come.
+If you encounter unexpected behavior, please post in Slack or open an issue.
+:::
+
+```bash
+dbt run -m state:modified --defer --state path/to/prod/artifacts
+dbt test -m state:modified
+```
+
+For more details, see the docs on [deferred runs](run#deferring-to-previous-run-state) and [the state selection method](node-selection/methods#the-state-method).
 
 ## Pro-tips for dbt Projects
 ### Limit the data processed when in development

diff --git a/website/docs/docs/guides/migration-guide/upgrading-to-0-17-0.md b/website/docs/docs/guides/migration-guide/upgrading-to-0-17-0.md
@@ -246,7 +246,7 @@ BigQuery:
 ## New and changed documentation
 
 **Core**
-- [`path:` selectors](model-selection-syntax#the-path-operator)
+- [`path:` selectors](node-selection/methods#the-path-method)
 - [`--fail-fast`](commands/run#failing-fast)
 - [as_text Jinja filter](dbt-jinja-functions/as_text)
 - [accessing nodes in the `graph` object](dbt-jinja-functions/graph)

diff --git a/website/docs/docs/guides/migration-guide/upgrading-to-0-18-0.md b/website/docs/docs/guides/migration-guide/upgrading-to-0-18-0.md
@@ -0,0 +1,82 @@
+---
+title: "Upgrading to 0.18.0"
+
+---
+
+### Resources
+
+- [Changelog](https://github.com/fishtown-analytics/dbt/blob/dev/marian-anderson/CHANGELOG.md)
+- [Discussion: Prerelease](https://discourse.getdbt.com/t/prerelease-v0-18-0-marian-anderson/1545)
+
+## Breaking changes
+
+Please be aware of the following changes in v0.18.0. While breaking, we do not expect these to affect the majority of projects.
+
+### Adapter macros
+
+- dbt only has access to adapter plugin macros from the currently-in-use adapter or one of its dependencies, rather than all installed adapters in the namespace.
+- `adapter_macro` is no longer a macro and will raise a deprecation warning. Use `adapter.dispatch` instead.
+
+### Data tests
+
+- Data tests are written as CTEs instead of subqueries. Adapter plugins for databases that don't support CTEs may need to override this behavior.
+
+### Python requirements
+- Upgraded `snowflake-connector-python` dependency to 2.2.10 and enabled the SSO token cache
+
+## New features
+
+For more details, see [new and changed documentation](#new-and-changed-documentation) below.
+
+:::info [β] Beta Features
+There are several pieces of net-new functionality in v0.18.0, with iterative improvements to come. If you encounter unexpected behavior, please post in Slack or open an issue.
+:::
+
+### Node selection
+- methods: `config`, `test_type`, `test_name`, `package`, [β] `state`
+- intersections
+- nth-parent/child
+- [β] version-controlled YAML selectors
+- [β] defer unselected node references to state defined by a previous run's artifacts
+
+### Adapter macros
+- `adapter.dispatch` replaces `adapter_macro`, with much greater flexibility
+- Schema tests are now defined via `dispatch`, such that non-core plugins
+can override schema test definitions
+
+### Docs
+- Include static assets (such as images) in auto-generated docs site
+- Improved resource search
+- Project-level overviews
+
+### Database-specific
+- Specify IAM profile when connecting to Redshift
+- Snowflake query tags at connection and model level
+- Impersonate a BigQuery service account when connecting via oauth
+- Adding policy tags to BigQuery columns
+- Configure time-to-live for BigQuery tables
+
+## New and changed documentation
+
+**Core**
+- [node selection syntax](node-selection/syntax)
+- [list (ls)](commands/list)
+- [deferring to previous run state](run#deferring-to-previous-run-state)
+- [adapter.dispatch](adapter#dispatch)
+- [`asset-paths` config](asset-paths) (also updated [dbt_project.yml](dbt_project.yml.md) and [description](description))
+- [flag for colorized logs](run#enable-or-disable-colorized-logs)
+- [`full_refresh` config](full_refresh)
+
+**Docs**
+- [project-level overviews](documentation#custom-project-level-overviews)
+
+**Redshift**
+- [`iam_profile`](redshift-profile#specifying-an-iam-profile)
+
+**Snowflake**
+- `query_tag` in [profile](snowflake-profile), [model config](snowflake-configs#query-tags)
+
+**BigQuery**
+- [`impersonate_service_account`](https://docs.getdbt.com/reference/warehouse-profiles/bigquery-profile#service-account-impersonation)
+- [`policy_tags`](bigquery-configs#policy-tags)
+- [`hours_to_expiration`](bigquery-configs#controlling-table-expiration)
diff --git a/website/docs/faqs/beta-release.md b/website/docs/faqs/beta-release.md
@@ -0,0 +1,4 @@
+---
+title: What is a beta release?
+---
+This is a chance to try out brand-new functionality. You get to start planning for use cases that the next minor version will unlock. We get to hear from you about unexpected behavior and nasty bugs, so that the release candidate has more polish and fewer surprises.
diff --git a/website/docs/faqs/bq-impersonate-service-account.md b/website/docs/faqs/bq-impersonate-service-account.md
@@ -0,0 +1,18 @@
+---
+title: Why would I want to impersonate a service account?
+---
+
+You may want your models to be built using a dedicated service account that has
+elevated access to read or write data to the specified project or dataset.
+Typically, this requires you to create a service account key for running under
+development or on your CI server. By specifing the email address of the service
+account you want to build models as, you can use [Application Default Credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default) or the
+service's configured service account (when running in GCP) to assume the identity
+of the service account with elevated permissions.
+
+This allows you to reap the advantages of using federated identity for developers
+(via ADC) without needing to grant individual access to read and write data
+directly, and without needing to create separate service account and keys for
+each user. It also allows you to completely eliminate the need for service
+account keys in CI as long as your CI is running on GCP (Cloud Build, Jenkins,
+GitLab/Github Runners, etc).
diff --git a/website/docs/faqs/build-one-seed.md b/website/docs/faqs/build-one-seed.md
@@ -8,6 +8,6 @@ $ dbt seed --select country_codes
 ```
 There is also an `--exclude` option.
 
-Check out more in the [model selection syntax](model-selection-syntax) documentation.
+Check out more in the [model selection syntax](node-selection/syntax) documentation.
 
 Prior to v0.16.0, there was no way to build one seed at a time.
diff --git a/website/docs/faqs/prerelease-docs.md b/website/docs/faqs/prerelease-docs.md
@@ -0,0 +1,4 @@
+---
+title: Why are there "prerelease" docs?
+---
+We want to give beta testers the information they need to try out new features, without adding confusion to the current docs site. This is served from a long-lived `next` branch of the docs repo. Feedback on prerelease docs is also welcome—use the "Edit this page" feature at the bottom.
diff --git a/website/docs/faqs/run-downstream-of-seed.md b/website/docs/faqs/run-downstream-of-seed.md
@@ -2,7 +2,7 @@
 title: How do I run models downstream of a seed?
 ---
 
-You can run models downstream of a seed using the [model selection syntax](model-selection-syntax), and treating the seed like a model.
+You can run models downstream of a seed using the [model selection syntax](node-selection/syntax), and treating the seed like a model.
 
 For example, the following would run all models downstream of a seed named `country_codes`:
 

diff --git a/website/docs/faqs/run-one-model.md b/website/docs/faqs/run-one-model.md
@@ -9,4 +9,4 @@ To run one model, use the `--models` flag (or `-m` flag), followed by the name o
 $ dbt run --models customers
 ```
 
-Check out the [model selection syntax documentation](https://docs.getdbt.com/docs/model-selection-syntax) for more operators and examples.
+Check out the [model selection syntax documentation](node-selection/syntax) for more operators and examples.
diff --git a/website/docs/faqs/run-one-snapshot.md b/website/docs/faqs/run-one-snapshot.md
@@ -9,4 +9,4 @@ To run one snapshot, use the `--select` flag, followed by the name of the snapsh
 $ dbt snapshot --select order_snapshot
 ```
 
-Check out the [model selection syntax documentation](model-selection-syntax) for more operators and examples.
+Check out the [model selection syntax documentation](node-selection/syntax) for more operators and examples.
diff --git a/website/docs/faqs/running-models-downstream-of-source.md b/website/docs/faqs/running-models-downstream-of-source.md
@@ -14,4 +14,4 @@ To run models downstream of one source table:
 $ dbt run --models source:jaffle_shop.orders+
 ```
 
-Check out the [model selection syntax](model-selection-syntax) for more examples!
+Check out the [model selection syntax](node-selection/syntax) for more examples!
diff --git a/website/docs/faqs/test-one-model.md b/website/docs/faqs/test-one-model.md
@@ -7,4 +7,4 @@ Running tests on one model looks very similar to running a model: use the `--mod
 dbt test --models customers
 ```
 
-Check out the [model selection syntax documentation](model-selection-syntax) for more operators and examples.
+Check out the [model selection syntax documentation](node-selection/syntax) for more operators and examples.
diff --git a/website/docs/faqs/test-sources.md b/website/docs/faqs/test-sources.md
@@ -8,4 +8,4 @@ Because the test selection syntax grew out of the model selection syntax (and pr
 $ dbt test --models source:*
 ```
 
-Check out the [model selection syntax documentation](model-selection-syntax#test-selection-examples) for more operators and examples.
+Check out the [model selection syntax documentation](node-selection/test-selection-examples) for more operators and examples.
diff --git a/website/docs/faqs/testing-sources.md b/website/docs/faqs/testing-sources.md
@@ -21,4 +21,4 @@ And, to run tests on one source table only:
 $ dbt test --models source:jaffle_shop.orders
 ```
 
-Yep, we know this syntax is a little less than ideal, so we're hoping to improve it in a future release. Check out the [model selection syntax](model-selection-syntax) for more examples!
+Yep, we know this syntax is a little less than ideal, so we're hoping to improve it in a future release. Check out the [model selection syntax](node-selection/syntax) for more examples!
diff --git a/website/docs/learn/3-build-your-first-models.md b/website/docs/learn/3-build-your-first-models.md
@@ -271,7 +271,7 @@ This time when dbt ran, separate views/tables were created for `stg_customers`,
 
 ## Extra exercises
 * Write some bad SQL to cause an error — can you debug this error?
-* Run only a single model at a time ([docs](model-selection-syntax))
+* Run only a single model at a time ([docs](node-selection/syntax))
 * Group your models with a `stg_` prefix into a `staging` subdirectory (i.e. `models/staging/stg_customers.sql`)
     * Configure your `staging` models to be views
     * Run only the `staging` models

diff --git a/website/docs/reference/commands/list.md b/website/docs/reference/commands/list.md
@@ -11,43 +11,55 @@ The `dbt ls` command lists resources in your dbt project. It accepts selector ar
 ```
 dbt ls
      [--resource-type {source,analysis,model,snapshot,test,seed,default,all}]
-     [--select SELECTOR [SELECTOR ...]]
+     [--select SELECTION_ARG [SELECTION_ARG ...]]
      [--models SELECTOR [SELECTOR ...]]
      [--exclude SELECTOR [SELECTOR ...]]
+     [--selector YML_SELECTOR_NAME [YML_SELECTOR_NAME ...]]
      [--output {json,name,path,selector}]
 ```
 
+See [resource selection syntax](node-selection/syntax) for more information on how to select resources in dbt
+
 **Arguments**:
 - `--resource-type`: This flag limits the "resource types" that dbt will return in the `dbt ls` command. By default, the following resources are included in the results of `dbt ls`: models, snapshots, seeds, tests, and sources.
-- `--select`: This flag specifies one or more "selectors" used to filter the nodes returned by the `dbt ls` command. See the docs on the [resource selection syntax](model-selection-syntax) for more information on selecting resources in dbt
+- `--select`: This flag specifies one or more selection-type arguments used to filter the nodes returned by the `dbt ls` command
 - `--models`: Like the `--select` flag, this flag is used to select nodes. It implies `--resource-type=model`, and will only return models in the results of the `dbt ls` command.
 - `--exclude`: Specify selectors that should be _excluded_ from the list of returned nodes.
+- `--selector`: This flag specifies one or more named selectors, defined in a `selectors.yml` file.
 - `--output`: This flag controls the format of output from the `dbt ls` command.
 
 Note that the `dbt ls` command does not include models which are disabled or schema tests which depend on models which are disabled. All returned resources will have a `config.enabled` value of `true`.
 
 ### Example usage
 
-**Listing models by selector**
+**Listing models by package**
 ```
 $ dbt ls --models snowplow.*
-model.snowplow.snowplow_base_events
-model.snowplow.snowplow_base_web_page_context
-model.snowplow.snowplow_id_map
-model.snowplow.snowplow_page_views
-model.snowplow.snowplow_sessions
+snowplow.snowplow_base_events
+snowplow.snowplow_base_web_page_context
+snowplow.snowplow_id_map
+snowplow.snowplow_page_views
+snowplow.snowplow_sessions
 ...
 ```
 
 **Listing tests by tag name**
 ```
 $ dbt ls --select tag:nightly --resource-type test
-model.my_project.orders
-model.my_project.order_items
-model.my_project.products
+my_project.schema_test.not_null_orders_order_id
+my_project.schema_test.unique_orders_order_id
+my_project.schema_test.not_null_products_product_id
+my_project.schema_test.unique_products_product_id
 ...
 ```
 
+**Listing schema tests of incremental models**
+```
+$ dbt ls --select config.materialized:incremental,test_type:schema
+model.my_project.logs_parsed
+model.my_project.events_categorized
+```
+
 **Listing JSON output**
 ```
 $ dbt ls --models snowplow.* --output json