dbt Metadata and State-Aware Operations — Part 3

Jeremiah Hansen

16 min read

·

Oct 6, 2026

--

Listen

Share

Introduction

Welcome to the third and final part in my dbt Metadata series! This is the blog post that I wanted to share to begin with, but I realized that for it to make sense I had to first lay the foundation for it — and that’s what I did in the first two parts. In the first part, I covered the essentials of dbt metadata, primarily focusing on dbt’s two main artifacts, the manifest.json and run_results.json. Then in the second part I covered the internals of Snowflake’s DBT PROJECT object and how it greatly simplifies the management of those important dbt artifacts for you. And now, in this final part, I will tie everything together and cover how to actually use these dbt artifacts to perform critical dbt operations such as state-aware deployments and orchestrations.

There are lots of terms floating around out there that are all related to using dbt artifacts — terms such as state (used in many different senses), state-aware, deferral, “slim ci”, and more. And over the years I have found that data engineers working with dbt are often confused by these terms, or haven’t become proficient working with dbt artifacts. But that’s about to change! My hope is that this blog post will clear up a great deal of that confusion, and get you started on using these powerful features to your advantage! So let’s get started.

For reference, here’s the outline of the three-part series:

1. dbt Metadata Essentials — Part 1
2. dbt Metadata and Snowflake dbt Projects — Part 2
3. dbt Metadata and State-Aware Operations — Part 3 (this blog post)

[This blog post was hand written.]

Two Modes of Operation

Before we get too deep into things, there are two topics we need to cover. The first is one that causes a lot of confusion among data engineers. dbt has two fundamental modes of operation that need to be kept straight — deploy and refresh. Here’s a diagram that helps make the split clear:

Press enter or click to view image in full size

The same dbt command line tool can be used to either create or update resources in an environment (deploy) or to refresh data in an environment (refresh). One helpful way to distinguish the two modes is by where they’re called from — deployments are called from a CI/CD pipeline while refreshes are called from an orchestration tool. There are different patterns for each that will be discussed in the sections below, but please keep this distinction in mind as we continue; it will be very helpful.

Node Selectors

The second thing we need to cover before we get too deep into things is dbt’s node selection syntax/language. It’s critical that you understand this. In part 1 I went into great detail about the core dbt artifacts manifest.json and run_results.json. A “node” in dbt represents the core supported object types, including models, tests, sources, seeds, snapshots, exposures, and analyses. The nodes in dbt are compiled into a Directed Acyclic Graph (or DAG), which captures the relationships between them all. Here’s a very simple example DAG:

Press enter or click to view image in full size

Once you have all the nodes connected together in a DAG, you need some way to select one or more of those nodes to act on them. And that’s exactly what node selectors are. dbt has defined a syntax for selecting nodes using the --select (or -s for short) and --exclude parameters. Here are a few simple examples using the DAG above:

• --select "node_b" – Selects node_b
• --select "node_b+" – Selects node_b, node_c and node_d
• --select "+node_d" – Selects node_a, node_b and node_d

Those are very simple examples, using only the name of the node and plus operator which selects ancestors or descendants. But dbt’s node selection syntax is incredibly robust, and allows you to select nodes in almost any manner conceivable. In addition to selecting by name, the other main approach is with node selector methods which follow this pattern: method:value. Here are a few examples:

• --select "resource_type:test" – Selects all test nodes in the project
• --select "tag:nightly" – Selects all nodes that have the tag nightly set
• --select "config.materialized:incremental" – Selects all nodes with an incremental materialization type

But that only scratches the surface of what’s possible with dbt’s node selector methods, and node selection syntax generally. Here’s a summary of the available node selector methods, in alphabetical order:

• access: – models by their access property. Values: public, private, protected.
• config: – nodes matching a specified config value, including booleans, dictionary keys, and values in lists (e.g. config.materialized:incremental, config.schema:audit, config.unique_key:column_a, config.meta.contains_pii:true).
• exposure: – parent resources of a specified exposure; use with the + operator.
• file: – a model or function by its filename, including the extension.
• fqn: – nodes by their fully qualified name (project name, subdirectories, and file name, period-separated).
• group: – models defined within a group.
• metric: – parent resources of a specified metric; use with the + operator.
• package: – nodes defined in the root project or an installed package; package:this selects the current project.
• path: – models, sources, or functions at or under a specific path.
• resource_type: – nodes of a particular type (model, test, exposure, function, source, etc.); similar to the --resource-type flag.
• result: – resources by their result status from a prior run (requires a prior run/test/build/seed and --state). Values: error, success, skipped (run/build/seed), fail, warn, pass (tests). Combine with + for upstream/downstream.
• saved_query: – saved queries.
• selector: – nodes defined by a named YAML selector in selectors.yml; composable with other methods, graph operators, and set operators.
• semantic_model: – semantic models.
• source: – source nodes by name; use with the + operator to find models that select from a specified source.
• source_status: – sources by freshness change versus a prior sources.json artifact (requires dbt source freshness in both states). Value: fresher.
• state: – nodes compared against a previous manifest (requires --state). Values: new, modified (plus inverses old, unmodified). Sub-selectors of modified: .body, .configs, .relation, .persisted_descriptions, .macros, .contract.
• tag: – nodes that match a specified tag.
• test_name: – tests based on the name of the generic test that defines them (e.g. unique, equality, or a custom test).
• test_type: – tests by type. Values: unit, data, generic, singular.
• unit_test: – unit tests by name.
• version: – versioned models by version status. Values: latest, prerelease, old, none.

Again, it’s critical that you understand dbt’s node selection syntax, so please check out dbt’s node selection documentation for all the details.

dbt and State

Alright, with that understanding in mind, let’s turn now to state-aware operations. There are three main categories of state in dbt Core:

• Object state — The state of a node (new, modified, old, unmodified)
• Execution state — The state of a node execution (error, success, skipped (run/build/seed); fail, warn, pass (tests))
• Source state — The state of data freshness in a source node (fresher)

And if you noticed, those three types of state correspond to three of the twenty-two node selector methods listed in the previous section: state, result, and source_status. Unfortunately the state method name is ambiguous and causes confusion as it only applies to object state, not the other types of state. While the remaining nineteen selector methods are very helpful for selecting nodes, these three methods together enable what I call state-aware operations.

But how exactly can dbt identify these various states? How does it know if a node (object) has been modified? Or how does it know if a previous node execution failed? Or if a source node has new data? The answer to this question gets right to the heart of state-aware operations. In dbt these various states are determined by looking at, and comparing, the dbt artifact files that we discussed in depth in part 1 — in particular the manifest.json, run_results.json, and sources.json artifacts. To help explain that further, please see this diagram:

Press enter or click to view image in full size

In order to perform state-aware operations dbt must have access to the artifacts from a previous execution of dbt (shown as prev-target in the diagram). Then depending on the type of state-aware operation you’re performing dbt looks at, or compares, various artifact files. Here’s how each state-aware category is determined:

• Object state — The state of a node (new, modified, old, unmodified) is determined by comparing the manifest.json files between the prev-target folder and the target folder.
• Execution state — The state of a node execution (error, success, skipped (run/build/seed); fail, warn, pass (tests)) is determined by looking at the run_results.json file in the prev-target folder (no comparison needed here).
• Source state — The state of data freshness in a source node (fresher) is determined by comparing the sources.json files between the prev-target folder and the target folder.

And the final part of the diagram to highlight here is the use of the --state argument. As with the state node selector method name, the name used here is unfortunately ambiguous. But the --state argument takes the path to the prev-target folder. That is how dbt knows where to look to find the previous artifact files.

Side Note: This is why I called out in part 1 about needing to have a strategy for preserving these files after each execution, because dbt needs them to perform state-aware operations. But like I mentioned there, dbt overwrites the artifacts in the /target folder during every invocation so it’s critical to first preserve the existing artifacts before they get replaced. Also, this is why I made such a big deal in part 2 about Snowflake’s DBT PROJECT object and how it automatically preserves the artifacts from previous executions in the results stage. Hopefully the pieces of the puzzle are coming together for you here.

So with all that knowledge, let’s now look at a few examples of what a state-aware operation looks like in dbt:

Object state (state selector method)

• dbt run --select "state:modified+" --state path/to/prev/target (Run all models that have been modified along with their downstream models)
• dbt test --select "state:new" --state path/to/prev/target (Run all new tests)

Execution state (result selector method)

• dbt run --select "result:error" --state path/to/prev/target (Run all models that failed during the previous run)
• dbt test --select "result:fail" --state path/to/prev/target (Run all tests that failed during the previous run)

Source state (source_status selector method)

• dbt build --select "source_status:fresher+" --state path/to/prev/target (Run all downstream models dependent on a source that has new data)

Hopefully that all makes sense, and if it’s not totally clear yet please go back and re-read this section. This part is the most important to internalize clearly, as the next, and final, layers build on top of this core dbt feature.

Pro Tip: One very helpful dbt command to be aware of is the list command (dbt ls or dbt list). The dbt list command lists resources in your project using the node selection and state-aware capabilities, without running any of the models. When testing out node selection, particularly in state-aware situations, it’s easiest to try them out with dbt list, like this: dbt ls --select "state:modified+" --state path/to/prev/target.

Advanced Example: Here’s an advanced example to show just how powerful dbt’s node selectors can be: dbt test --select "state:modified+1 1+exposure:*,state:modified+". In this example state:modified+1 refers to the modified model and its first-order children. 1+exposure:*,state:modified+ uses the intersection operator to get any models which are referenced in an exposure and depend on a modified model. Wow! Hopefully you can see just how powerful the node selectors are.

Snowflake dbt Projects and State

We’re almost there! Building on everything we’ve discussed so far (in this entire blog series), I’m now going to show how to make state-aware operations a reality. This section will show how to access and make use of the dbt artifacts that are automatically maintained by Snowflake’s dbt Projects capability. This might be a good time to go back to part 2 and review how the results stage and results stage helper functions work — that mechanism for accessing previous artifacts will be foundational here.

The first thing we need to cover is how the IMPORTS clause works in Snowflake’s dbt Projects feature. This clause only appears in the EXECUTE DBT PROJECT statement, which is defined as:

EXECUTE DBT PROJECT [ IF EXISTS ] <name>
  [ ARGS = '[ <dbt_command> ] [ --<dbt_cli_option> <option_value_1> [ ... ] ] [ ... ]' ]
  [ DBT_VERSION = 'version_number' ]
  [ EXTERNAL_ACCESS_INTEGRATIONS = ( <integration_name> [ , ... ] ) ]
  [ ENVIRONMENT = '<environment_name>' ]
  [ ENV_VARS = ( '<key>' = '<value>' [ , ... ] ) ]
  [ IMPORTS = ( { '<source_location>' | <system_function> } [ AS '<alias>' ] [ , ... ] ) ]
  [ WRITEBACK = { TRUE | FALSE } ]

During a Snowflake dbt Project execution, a sandbox execution environment is created in the Snowflake warehouse. And before the actual dbt execution, Snowflake copies all the required files for the execution into the sandbox so that they are all available to dbt during execution. The first thing that is copied into the sandbox is the entire contents of the live stage (revisit in part 2) which includes the core dbt project files, and any dependent packages and compiled code there may be. All that is left to add to the sandbox before executing are any previous artifacts. And those are added to the sandbox with the IMPORTS clause. Here’s a diagram showing the process I just described:

Press enter or click to view image in full size

The IMPORTS clause is a very powerful and flexible feature that allows you to copy sets of artifact files from a number of different source locations. To review, here’s the syntax of the IMPORTS clause again:

IMPORTS = ( { '<source_location>' | <system_function> } [ AS '<alias>' ] [ , ... ] )

There are a number of different types of source locations supported, and each one listed will be copied into its own folder in the /imports folder of the sandbox. The folder name can be controlled with the <alias> clause. Here’s a quick overview of the types of source locations that are supported:

• dbt Project results stage
• Another dbt Project object
• ZIP archive file (from dbt Project results stage)
• Internal Stage
• Workspace
• Git Repository

The IMPORTS clause expects a source location path which can be specified directly, or which can be dynamically generated with the help of one of the results stage helper functions. For help with understanding the structure of the source location paths, please check out my Snowflake dbt Projects Operations Deep Dive blog post. And for help with understanding the results stage helper functions, please see part 2 or the Snowflake Access dbt artifacts and logs programmatically documentation page for more information.

After the dbt execution is complete, but before the EXECUTE DBT PROJECT command returns, Snowflake automatically preserves the dbt artifacts. This step is ultimately what enables everything that is being discussed here, the state-aware operations. As I covered in the results stage section in part 2, Snowflake automatically copies the target/ and logs/ folders to the results stage for future use. In addition to that, if WRITEBACK is set to TRUE (the default) then Snowflake will additionally copy the target/ and logs/ folders back into the live stage. See the sandbox diagram above for the complete lifecycle.

State-Aware Operations in Snowflake

To put a final bow on the discussion, I will give a few examples of actually executing state-aware operations in Snowflake with Snowflake dbt Projects. Here is the canonical pattern that combines everything we’ve discussed up until now:

EXECUTE DBT PROJECT my_db.my_schema.my_dbt_project
  ARGS = 'run --state ./imports/prev-target --select "state:modified+"'
  IMPORTS = (
    SYSTEM$DBT_GET_LAST_SUCCESSFUL_RUN_TARGET(
      'my_db.my_schema.my_dbt_project'
    ) AS 'prev-target'
  );

There is a LOT happening in that single statement, which you should now understand. To recap, it uses IMPORTS to copy the dbt artifacts from the previous execution via the results stage and results stage helper function SYSTEM$DBT_GET_LAST_SUCCESSFUL_RUN_TARGET(), and then executes dbt (hosted entirely by Snowflake in the warehouse) with the --state flag pointing to the copy of the previous target folder in the sandbox and finally uses a state-aware object state selector (state:modified+) to select only modified models and their downstream models, which dbt accomplishes by comparing the manifest.json artifact file between the prev-target folder and the current target folder!

Take a second to let that soak in. That SQL EXECUTE DBT PROJECT statement is incredibly powerful, so don’t miss it. By automatically maintaining previous copies of the dbt artifacts in the results stage, Snowflake’s dbt Project feature makes that very complex operation as simple as possible while giving you complete control over the process!

While the example I gave above represents the canonical pattern with Snowflake’s dbt Projects capability, it is a relatively straightforward example. There are lots of different variations that can be created from that example to cover any situation you run into. By adapting the pattern above you can enable any state-aware operation supported by dbt Core in Snowflake dbt Projects, be that for deploying or refreshing dbt resources.

Please also note that by default the SYSTEM$DBT_GET_LAST_*_RUN_TARGET() functions only search compile, build, run, docs generate runs, so if you want to perform state-aware operations based on tests or source freshness, you need to explicitly pass test or source freshness as the second argument to those functions. For more details and examples check out part 2 and the Snowflake EXECUTE DBT PROJECT documentation.

Working in Sparse Environments

One major source of confusion in this whole discussion is between state-aware operations (as described above) and working with sparse environments (described here). dbt has features to support working in sparsely populated environments, primarily the defer feature with --defer. You’ll notice that in the entire discussion above I didn’t mention deferring or --defer once, and that was intentional. Most discussions on dbt state-aware operations get mixed up with ones on working in sparse environments, as if the two must always happen together. But as shown above many important scenarios can be covered without deferrals.

All that said, working in sparse environments with --defer is a common and powerful feature in dbt, and worth a brief discussion here. First, let me define a sparse environment. A sparse environment is a pre-production environment (dev, integration, stage, etc.) that does not contain a complete set of objects (tables, views, etc.). This happens often in integration environments (during a CI process) where the user wants to quickly stand up an environment and test a few changed dbt models, but doesn’t want to have to create an entire copy of the environment. This particular situation is referred to by dbt as “slim ci”. But that name is a bit misleading as there are other scenarios that are supported by sparse environments, such as local development, hotfixes, or even as a way to manage other pre-production environments.

Let me explain how deferral works. Imagine you have a very simple dbt project, with only three models connected in order — model_a -> model_b -> model_c. And imagine that you have an empty development environment. If you made a change to model_c and want to test it, you might try and run a dbt command like dbt build --select "+model_c", which would run every model in the project, and fully populate the development environment like this:

Press enter or click to view image in full size

That might be fine for an environment as simple as this, but in a real environment you will have a lot of models and trying to re-run all of them, just to test the one new model, is very wasteful (in both time and money). Instead, what if you could only create model_c in your development environment and simply refer to model_b in a different environment (production or another fully populated pre-production environment)? That’s exactly what deferral does in dbt. Here’s a visual representation of that approach:

Press enter or click to view image in full size

And dbt makes this possible with the --defer argument, used like this:

dbt build --state path/to/prod/artifacts --defer --select "model_c"

For this example to work, deferring to the production environment, dbt needs to have access to the artifacts from a previous run against the production environment. dbt needs the production manifest.json file to determine the exact location of the production models so it can repoint the references in the model_c code running in development against the production source models. defer only rewrites references to upstream nodes that aren't selected (and, unless --favor-state is set, that don't exist in dev). By default --defer will look to the artifacts identified by the --state argument, but dbt also supports a separate argument --defer-state for situations where you need two sets of artifacts, one for deferral and the other for the state-aware operations.

Snowflake’s dbt Projects capability supports deferral with the mechanisms previously described. The only difference with this pattern is that for deferral you need to refer to dbt artifacts from an execution of dbt from a different environment (production in the example used here). Notice the two different database names in this example with Snowflake’s dbt Projects (my_dev_db and my_prod_db):

EXECUTE DBT PROJECT my_dev_db.my_schema.my_dbt_project
  ARGS = 'run --state ./imports/prev-target --defer --select "state:modified+"'
  IMPORTS = (
    SYSTEM$DBT_GET_LAST_SUCCESSFUL_RUN_TARGET(
      'my_prod_db.my_schema.my_dbt_project'
    ) AS 'prev-target'
  );

But as I said earlier, this is made incredibly easy with Snowflake’s dbt Projects capability since Snowflake automatically captures the previous run artifacts in the results stage of the object!

Conclusion

When I first shared part 1 of this blog series on LinkedIn I asked the question, “on a scale of 1 to 10, how would you rate your knowledge of, and proficiency working with, dbt metadata?” I’m not sure how you would’ve rated yourself prior to finishing the series, but I’ll ask you now, after going through this three-part series how would you rate your knowledge of, and proficiency working with, dbt metadata, on a scale of 1 to 10?

While you may not have had time to put this all into practice yet, I hope at least your knowledge of dbt metadata and state-aware operations is way up. And I hope that I’ve opened the door for you to begin quickly implementing advanced state-aware operations with Snowflake’s dbt Projects feature!

Good luck! And if you end up implementing any of these patterns with Snowflake dbt Projects, I’d love to hear about it — come find me on LinkedIn and let me know how it went.

Blog posts in this series:

1. dbt Metadata Essentials — Part 1
2. dbt Metadata and Snowflake dbt Projects — Part 2
3. dbt Metadata and State-Aware Operations — Part 3 (this blog post)