Skip to content

Commit

Permalink
Fix/remove smart failure reruns language (dbt-labs#1308)
Browse files Browse the repository at this point in the history
* remove smart failure reruns language

* fix (most) broken links

Co-authored-by: Leona B. Campbell <[email protected]>
  • Loading branch information
matt-winkler and runleonarun authored May 12, 2022
1 parent a707d14 commit ef6aec1
Show file tree
Hide file tree
Showing 6 changed files with 16,885 additions and 75 deletions.
12 changes: 6 additions & 6 deletions website/blog/2019-05-01-how-we-structure-dbt-projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ It’s important to note that **this is not the only, or the objectively best, w

Our opinions are **almost guaranteed to change over time** as we update our views on modeling, are exposed to more analytics problems, and data stacks evolve. It’s also worth clearly stating here: the way we structure dbt projects makes sense for our projects, but may not be the best fit for yours! This article exists on Discourse so that we can have a conversation – I would love to know how others in the community are structuring their projects.

In comparison, the (recently updated) [best practices](/docs/best-practices) reflect principles that we believe to be true for any dbt project. Of course, these two documents go hand in hand – our projects are structured in such a way that makes the those principles easy to observe, in particular:
In comparison, the (recently updated) [best practices](/docs/guides/best-practices) reflect principles that we believe to be true for any dbt project. Of course, these two documents go hand in hand – our projects are structured in such a way that makes the those principles easy to observe, in particular:

* Limit references to raw data
* Rename and recast fields once
Expand Down Expand Up @@ -104,9 +104,9 @@ Each staging directory contains at a minimum:
* Named `stg_<source>__<object>`.
* Generally materialized as a <Term id="view" /> (unless performance requires it as a table).
* A `src_<source>.yml` file which contains:
* [Source](/docs/using-sources) definitions, tests, and documentation
* [Source](/docs/building-a-dbt-project/using-sources) definitions, tests, and documentation
* A `stg_<source>.yml` file which contains
* [Tests and documentation](/docs/testing-and-documentation) for models in the same directory
* [Tests](/docs/building-a-dbt-project/tests) and [documentation](/docs/building-a-dbt-project/documentation) for models in the same directory

```
├── dbt_project.yml
Expand All @@ -127,7 +127,7 @@ Some dbt users prefer to have one `.yml` file per model (e.g. `stg_braintree__cu

Earlier versions of the dbt documentation recommended implementing “base models” as the first layer of transformation – and we used to organize and name our models in this way, for example `models/braintree/base/base_payments.sql`.

We realized that while the reasons behind this convention were valid, the naming was an opinion, so in our recent update to the [best practices](/docs/best-practices), we took the mention of base models out. Instead, we replaced it with the principles of “renaming and recasting once” and “limiting the dependencies on raw data”.
We realized that while the reasons behind this convention were valid, the naming was an opinion, so in our recent update to the [best practices](/docs/guides/best-practices), we took the mention of base models out. Instead, we replaced it with the principles of “renaming and recasting once” and “limiting the dependencies on raw data”.

That being said, in our dbt projects every source flows through exactly one model of the following form:

Expand Down Expand Up @@ -210,7 +210,7 @@ Where the work of staging models is limited to cleaning and preparing, fact tabl
This layer of modeling is considerably more complex than creating staging models, and the models we _design_ are highly tailored to the analytical needs of an organization. As such, we have far less convention when it comes to these models. Some patterns we’ve found to be useful are:

* `fct_` and `dim_` models should be materialized as tables within a warehouse to improve query performance. As a default, we use the table materialization, and where performance requires it, we use the incremental materialization.
* Intermediate transformations required to get to a fact or dimension model are placed in a nested `marts/<mart>/intermediate` directory. They are named `<useful_name>__<transformation_in_past_tense>.sql`. The lack of prefix and use of double underscores indicates that these are intermediate models, not to be trusted, however, it may also be worth hiding these in a different [schema](/docs/using-custom-schemas).
* Intermediate transformations required to get to a fact or dimension model are placed in a nested `marts/<mart>/intermediate` directory. They are named `<useful_name>__<transformation_in_past_tense>.sql`. The lack of prefix and use of double underscores indicates that these are intermediate models, not to be trusted, however, it may also be worth hiding these in a different [schema](/docs/building-a-dbt-project/building-models/using-custom-schemas).
* Models are tested and documented in a `<dir_name>.yml` file in the same directory as the models.
* Any extra documentation in a [docs block](/docs/building-a-dbt-project/documentation#using-docs-blocks) is placed in a `<dir_name>.md` file in the same directory.

Expand Down Expand Up @@ -251,7 +251,7 @@ There are other kinds of SQL files that find their way into robust dbt projects.
* `metrics`**:** Precisely defined measurements taken from fact tables, directly conducive to time-series reporting, and tightly structured so as to allow one-to-one comparison with goals and forecasting. A metrics table lives downstream of dimension and fact tables in your DAG, and it deserves special status.
* **Packages:** While not a model folder within your main project, packages that include models (like our [snowplow](https://github.com/dbt-labs/snowplow) package) can be configured into custom schema and materialization patterns from `dbt_project.yml`.

In projects where we find ourselves with these additional models, we often leverage [custom schemas](/docs/using-custom-schemas) as directories in our warehouse, to logically group the models, choosing a schema name that matches the directory name in our dbt project.
In projects where we find ourselves with these additional models, we often leverage [custom schemas](/docs/building-a-dbt-project/building-models/using-custom-schemas) as directories in our warehouse, to logically group the models, choosing a schema name that matches the directory name in our dbt project.

## Final thoughts
-----------------------------------
Expand Down
4 changes: 2 additions & 2 deletions website/blog/2021-02-05-dbt-project-checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ This post is the checklist I created to guide our internal work, and I’m shari

**Useful links**:

* [.yml files](/docs/building-a-dbt-project/testing-and-documentation/schemayml-files/#usage)
* [.yml files](/docs/building-a-dbt-project/using-sources/#testing-and-documenting-sources)
* [Materializations](/docs/building-a-dbt-project/building-models/materializations/#configuring-materializations)
* [YAML selectors](/reference/node-selection/yaml-selectors/)

Expand Down Expand Up @@ -200,7 +200,7 @@ Are you using the IDE and if so, how well?

**Useful links**

* [dbt Cloud as a CI tool](/docs/dbt-cloud/using-dbt-cloud/cloud-enabling-continuous-integration-with-github/)
* [dbt Cloud as a CI tool](/docs/dbt-cloud/using-dbt-cloud/cloud-enabling-continuous-integration-with-github)


## ✅ DAG Auditing
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ clean-targets:
- Do you have custom scripts that parse dbt JSON artifacts?
- (BigQuery only) Do you use dbt's legacy capabilities around ingestion-time-partitioned tables?

If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-1-0-0).
If you believe your project might be affected, read more details in the migration guide [here](/docs/guides/migration-guide/upgrading-to-v1.0).

</details>

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ The example query returns information about all snapshots in this job.
```

### Fields
Snapshots has access to the *same fields* as the [Snapshot node](/docs/dbt-cloud/dbt-cloud-api/metadata/schema/metadata-schema-snapshot). The difference is that Snapshots can output a list, so instead of querying for fields for one specific snapshot, you can query for those parameters for all snapshots within a jobID, database, etc.
Snapshots has access to the *same fields* as the [Snapshot node](/docs/dbt-cloud/dbt-cloud-api/metadata/schema/metadata-schema-snapshots). The difference is that Snapshots can output a list, so instead of querying for fields for one specific snapshot, you can query for those parameters for all snapshots within a jobID, database, etc.

When querying for `snapshots`, the following fields are available:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,17 +67,6 @@ Because dbt Cloud manages deferral and state environment variables, there is no

To learn more about state comparison and deferral in dbt, read the docs on [state](understanding-state).

## Smart Reruns

As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that failed and had errors.

When a job is selected, dbt Cloud will surface the artifacts from that job's most recent successful run. dbt will then use those artifacts to determine the set of error/fail resources. In your job commands, you can signal to dbt to run and test only on these error/fail results and their children by including the `result:error+` and `result:fail+` argument.

As example:
```bash
dbt build --select result:error+ result:fail+
```

## Fresh Rebuilds

As an extension of the Slim CI feature, dbt Cloud can rerun and retest only the things that are fresher compared to a previous run.
Expand Down Expand Up @@ -106,7 +95,7 @@ dbt build --select source_status:fresher+
```
</VersionBlock>

More example commands in [Pro-tips for workflows](/docs/guides/best-practices.md#pro-tips-for-workflows)
More example commands in [Pro-tips for workflows](/docs/guides/best-practices.md#pro-tips-for-workflows).

## Troubleshooting

Expand Down
Loading

0 comments on commit ef6aec1

Please sign in to comment.