dbt Metadata Essentials — Part 1

Jeremiah Hansen

12 min read

·

Sep 12, 2026

--

2

Listen

Share

Introduction

This series is all about dbt metadata. It is written for data engineers who are building their data pipelines with dbt, and who want to better understand and work with dbt metadata. Properly leveraging dbt metadata is critical to success with dbt, because it’s the foundation for critical operational activities, including:

• Monitoring dbt’s executions
• Implementing state-aware deployments and orchestration
• Integrating technical and business metadata
• And more

The first dbt metadata use case most people start with is monitoring and analyzing execution results and logs. And that’s where we’ll start as well. This blog post is the first in a series of three posts that will cover the essentials of dbt metadata and how to effectively use it with Snowflake’s dbt Projects feature. Here’s the outline of the three-part series:

1. dbt Metadata Essentials — Part 1 (this blog post)
2. dbt Metadata and Snowflake dbt Projects — Part 2
3. dbt Metadata and State-Aware Operations — Part 3

Let’s begin by taking a deeper look at dbt’s metadata.

[This blog post was hand written.]

dbt Artifacts Overview

Every time you execute the dbt CLI, one or more artifact files are produced. These artifacts can be found in the /target folder (the default location that can be overridden), and they are written in the JSON format. There are around 10 different types of artifacts that can be produced, but the two most important by far are these:

• manifest.json: This file contains a full representation of your dbt project's resources (models, tests, macros, and more), including all node configurations and resource properties. (from manifest.json docs)
• run_results.json: This file contains information about a completed invocation of dbt, including timing and status info for each node (model, test, etc) that was executed. (from run_results.json docs)

The next sections will go into more detail on those two particular artifacts. But I will also mention here a few other helpful artifacts to be aware of:

• sources.json: This file contains information about sources with freshness checks. (from sources.json doc)
• osi_document.json: This file represents your project's Semantic Layer in the Apache Ossie format, a vendor-agnostic schema for describing semantic models and metrics. (from osi_document.json docs)
• catalog.json: This file contains information from your data warehouse about the tables and views produced and defined by the resources in your project. (from catalog.json docs)

For more information on these, and other, artifacts please refer to the core dbt artifacts and other dbt artifacts pages.

This next part is important. dbt overwrites the artifacts in the /target folder during every invocation! So you will need to have a strategy for preserving these files after each execution, so that they can be used later for various dbt operations. I will mention a few different use cases that leverage previous artifacts in the sections below. But for now, the important thing to know is that you have to have a strategy for preserving dbt artifacts. And here are a few such strategies:

• Copying selected (or all) artifact files from the /target folder to a unique archive location after execution
• Using the --target-path command line parameter to store the artifact files to a unique archive location
• Using the --no-write-json command line parameter to skip writing any artifact files during the execution, leaving previous files intact
• Using Snowflake’s dbt Projects feature, which automatically manages the artifacts for you! (more to come on this in parts 2 and 3)

We’ll now explore the two most important artifacts, manifest.json and run_results.json, in more detail. In short, the manifest.json describes what exists in the project – every model, test, source, macro, and the relationships between them – while the run_results.json records what happened during a run.

manifest.json Deep Dive

The manifest.json artifact contains a complete, static snapshot of your dbt project's structure at the time of compilation. It’s dbt’s complete self-description. The artifact is generated by almost all dbt commands, any that need to parse the project (triggering a compilation).

The manifest.json is used for a multitude of different purposes, including:

• dbt’s core execution engine (running models, tests, and more)
• State aware deployments and “Slim CI” (deploying only changed models, deferring to another environment for pre-prod testing, and more)
• Integration with third party tools for data cataloging, BI, semantic layers, documentation, and more
• Any other custom metadata extraction, integration or automation scenario

Remember: dbt overwrites the manifest.json file at the start of every invocation, during parsing! You will need to archive this file after execution so that it can be used in subsequent dbt runs. Please see the section above on dbt Artifacts for some archiving strategies.

Here’s a high-level, simplified diagram of the structure of the manifest.json:

Press enter or click to view image in full size

Along with a brief description of the top level keys in the file which give you a detailed understanding of what’s contained in the file (from the manifest.json docs):

• metadata: Info about the project and dbt environment
• nodes: Dictionary of all analyses, models, seeds, snapshots, and tests
• sources: Dictionary of sources
• metrics: Dictionary of metrics
• exposures: Dictionary of exposures
• groups: Dictionary of groups
• macros: Dictionary of macros
• docs: Dictionary of docs blocks
• parent_map: Dictionary that contains the first-order parents of each resource
• child_map: Dictionary that contains the first-order children of each resource
• group_map: Dictionary that maps group names to their resource nodes
• selectors: Expanded dictionary representation of YAML selectors
• disabled: Array of resources with enabled: false

Let’s quickly look at a few specific items to give you a better understanding of what is captured in the manifest.json. First the file-level metadata object, which contains information about the project and dbt environment:

{
  "dbt_version": "1.10.15",
  "generated_at": "2026-07-07T04:25:48Z",
  "invocation_id": "0c503aa2-0aa4-445c-83aa-ee8ed0c5a9f8",
  "project_name": "de214_demo",
  "project_id": "12c64d9ec1effbce8bf55b1134e2ac39",
  "adapter_type": "snowflake"
}

Notice the invocation_id here, it is important. invocation_id is a UUID assigned to the particular invocation (or execution) of the dbt CLI. This value is stored in many places, including the dbt artifacts and dbt logs, and can be used to connect the various pieces of metadata.

Next let’s take a look at one of the model nodes (notice the resource_type of “model”):

{
  "unique_id": "model.de214_demo.stg_customers",
  "resource_type": "model",
  "name": "stg_customers",
  "description": "Cleaned customer master with friendly names.",
  "language": "sql",
  "path": "staging/stg_customers.sql",
  "database": "DE214_DEMO",
  "schema": "DEV",
  "depends_on": {
    "nodes": ["source.de214_demo.raw.GEO", "source.de214_demo.raw.C_MST"]
  },
  "config": { "materialized": "table", "..." : "..." },
  "columns": {
    "customer_id": { "name": "customer_id", "description": "Unique customer identifier." }
  },
  "raw_code": "{{ config(materialized='table') }}\n\nwith customers as (\n    select\n        c_k    as customer_id,\n        c_nm   as customer_name,\n        c_seg  as market_segment,\n        c_nat  as nation_id,\n        c_abal as account_balance,\n        c_ph   as phone\n    from {{ source('raw', 'C_MST') }}\n),\ngeo as (\n    select\n        n_k  as nation_id,\n        n_nm as nation,\n        r_nm as region\n    from {{ source('raw', 'GEO') }}\n)\nselect\n    c.customer_id,\n    c.customer_name,\n    c.market_segment,\n    g.nation,\n    g.region,\n    c.account_balance,\n    c.phone\nfrom customers c\nleft join geo g\n    on c.nation_id = g.nation_id",
  "compiled_code": "with customers as (\n    select\n        c_k    as customer_id,\n        c_nm   as customer_name,\n        c_seg  as market_segment,\n        c_nat  as nation_id,\n        c_abal as account_balance,\n        c_ph   as phone\n    from DE214_DEMO.RAW.C_MST\n),\ngeo as (\n    select\n        n_k  as nation_id,\n        n_nm as nation,\n        r_nm as region\n    from DE214_DEMO.RAW.GEO\n)\nselect\n    c.customer_id,\n    c.customer_name,\n    c.market_segment,\n    g.nation,\n    g.region,\n    c.account_balance,\n    c.phone\nfrom customers c\nleft join geo g\n    on c.nation_id = g.nation_id","...": "21 more fields omitted (checksum, fqn, alias, tags, meta, patch_path, build_path, contract, primary_key, etc.)"
}

Here we find the full representation of the dbt model (stg_customers in this example) along with the model code. You’ll notice that the raw_code attribute contains the raw model code including all un-rendered jinja, while the compiled_code attribute contains the compiled model code with all configurations and jinja resolved.

Finally, let’s look at one of the test nodes (notice the resource_type of “test”) related to the above model:

{
  "unique_id": "test.de214_demo.unique_stg_customers_customer_id.c7614daada",
  "resource_type": "test",
  "test_metadata": {
    "name": "unique",
    "kwargs": { "column_name": "customer_id" }
  },
  "attached_node": "model.de214_demo.stg_customers",
  "config": {
    "severity": "ERROR",
    "warn_if": "!= 0",
    "error_if": "!= 0"
  },
  "raw_code": "{{ test_unique(**_dbt_generic_test_kwargs) }}",
  "compiled_code": "select\n    customer_id as unique_field,\n    count(*) as n_records\n\nfrom DE214_DEMO.DEV.stg_customers\nwhere customer_id is not null\ngroup by customer_id\nhaving count(*) > 1",
  "...": "16 more fields omitted (checksum, fqn, path, depends_on, column_name, extra_ctes, etc.)"
}

Here again we find the full representation of the dbt test (unique column test for stg_customers in this example) along with the test code.

Since the manifest.json file contains the complete, static snapshot of your dbt project's structure at the time of compilation, there is a lot more that could be said about this file. But this should give you a workable knowledge of this critical dbt artifact.

run_results.json Deep Dive

The run_results.json artifact contains a snapshot of what happened during a specific run/invocation — which nodes executed, whether they succeeded, and how long they took. It only contains details for the nodes that were actually executed during the run (models, tests, etc.) and not for all nodes in the project. The artifact is generated by almost all dbt commands.

The run_results.json is used for a multitude of different purposes, including:

• Integration with third party tools for job monitoring and observability
• State aware job orchestration (re-running from previous failure, retrying failed nodes, and more)
• Any other custom metadata extraction, integration or automation scenario that involves working with dbt execution results (including custom data engineering tools and frameworks)

Remember: dbt overwrites the run_results.json file at the end of every invocation! You will need to archive this file after execution so that it can be used in subsequent dbt runs. Please see the section above on dbt Artifacts for some archiving strategies.

Here’s a high-level, simplified diagram of the structure of the run_results.json:

Press enter or click to view image in full size

Along with a brief description of the top level keys in the file which give you a detailed understanding of what’s contained in the file (from the run_results.json docs):

• metadata: Info about the run itself
• args: Info about every CLI flag and setting dbt used
• elapsed_time: Total invocation time in seconds
• results: Array of node execution details

Let’s quickly look at a few specific items to give you a better understanding of what is captured in the run_results.json. First the file-level metadata object, which contains information about the run itself:

{
  "dbt_version": "1.10.15",
  "generated_at": "2026-07-07T04:26:15Z",
  "invocation_id": "0c503aa2-0aa4-445c-83aa-ee8ed0c5a9f8",
  "invocation_started_at": "2026-07-07T04:25:48Z"
}

Again, notice the invocation_id here, it is important. invocation_id is a UUID assigned to the particular invocation (or execution) of the dbt CLI, so it will be the same across the manifest.json, run_results.json, and log files for a particular invocation. You can use it to join these files together.

Next let’s look at the args object which provides critical context for this invocation:

{
  "invocation_command": "dbt run --target dev --profiles-dir /Users/john/.dbt",
  "which": "run",
  "target": "dev",
  "project_dir": "/Users/john/Documents/dbt_project",
  "select": [],
  "exclude": [],
  "vars": {},
  "defer": false,
  "favor_state": false,
  "partial_parse": true,
  "full_refresh": false,
  "warn_error_options": {
    "error": [],
    "warn": [],
    "silence": []
  },
  "...": "40+ more fields omitted (logging verbosity, telemetry, strict-mode/validation flags, caching behavior, etc.)"
}

The args object is dbt's snapshot of every CLI flag and global config value (whether explicitly passed or defaulted) at invocation time. This information is useful for reproducing a run exactly, or auditing which flags were active for a given result set. Of particular note is the invocation_command attribute which captures exactly how the dbt CLI was invoked.

Last, but certainly not least, we need to take a look at the heart and soul of the file, the results array. Here is a sample of one result object in the array:

{
  "unique_id": "model.de214_demo.stg_customers",
  "status": "success",
  "execution_time": 2.39,
  "timing": [
    { "name": "compile", "started_at": "...", "completed_at": "..." },
    { "name": "execute", "started_at": "...", "completed_at": "..." }
  ],
  "adapter_response": {
    "code": "SUCCESS",
    "rows_affected": 1,
    "query_id": "01c58aa9-0917-a118-0000-7111063966f2"
  },
  "failures": null,
  "compiled_code": "select c.customer_id, ...",
  "relation_name": "DE214_DEMO.DEV.stg_customers",
  "...": "3 more fields omitted (thread_id, message, compiled, batch_results)"
}

Here we find the detailed results of each node executed by dbt during this particular invocation (in this case the stg_customers model). It contains the execution status and all related timings, along with the adapter (database) results. The unique_id is the key that uniquely identifies this dbt node, which can be used to join back to the manifest.json (along with the invocation_id) to get the complete details about this node.

dbt Logs

In addition to dbt artifacts, there is one other important source of metadata generated by dbt that is worth mentioning — dbt logs. Events in dbt are written in real-time to two different places:

• The command line console (stdout, for interactive feedback)
• The log file (/logs/dbt.log, for offline debugging)

For the sake of this post we’ll focus on the log file itself. And compared with the dbt artifacts that we looked at above (particularly the manifest.json and run_results.json artifacts), the dbt log contains the following unique information:

• Details about every query sent to the warehouse including raw SQL text and execution details
• Detailed warehouse connection-pool lifecycle events
• Detailed information about the process/host (OS, arch) where the execution took place
• Detailed information on node execution, including cache/skip causation during execution
• Other dbt specific information and telemetry details

Of those items unique to the dbt logs, the most important is the first one. The dbt log captures details about every query sent to the warehouse including raw SQL text and execution details. This can be really important as the artifacts only contain the single/primary SQL query used by the model, and not every query executed by dbt during the materialization process. So the log (or query history in Snowflake) is the place to go to see every query issued by dbt during an execution.

A few last remarks about dbt logs. First, as I mentioned above, by default the dbt artifacts are written to the /target folder using the JSON format. With logs, the default is to write the dbt.log file to the /logs folder using the text format. But the format for the logs is configurable and can be set to either text or JSON. And second, dbt provides two sets of parameters/configurations for logs which allow for configuring the console logs independent from the log file:

• Console: --log-level, --log-format
• Log file: --log-level-file, --log-format-file, --log-path

To summarize, logs tell you what dbt did, step by step, against the target warehouse (every connection, every query, every skip reason) — artifacts tell you what the project is and what the final result was (the compiled SQL, the graph, the outcome per node). They’re complementary, not overlapping. And as mentioned above, the way to join (or correlate) the logs to the artifacts is with the shared invocation_id and per-node unique_id keys.

For more details, see dbt’s Events and Logs page.

Ingesting dbt Artifacts

A core capability of any enterprise data engineering tool (or framework) is to be able to log execution results for use in monitoring and operational activities. At a minimum those results should include details about every pipeline/activity that was run, including its result status and execution time. As we’ve seen with dbt, the manifest.json and the run_results.json artifacts contain exactly that metadata. So as you can imagine, it’s very common to want to ingest this important dbt metadata into persistent tables in Snowflake, to aid in monitoring.

There are two primary approaches to ingesting the dbt artifact metadata into persistent tables in Snowflake. At first glance, the most obvious way is to create a data ingestion process that reads the manifest.json and run_results.json directly and loads the data from there. But there is a more popular second way which is to create a process within dbt that reads from dbt’s internal data structures, which contain the same metadata, and load the data directly from there. This is generally accomplished by using a dbt on-run-end hook and calling a custom macro to do that work.

You could create such a process yourself (as just described), but thankfully there are others who have already done it and have shared their code. The two most popular options that I’m aware of are:

• dbtartifacts by Brooklyn Data (and link to GitHub repo dbtartifacts)
• elementary by Elementary (and link to GitHub repo dbt-data-reliability)

Both are implemented as dbt packages and hosted in dbt Hub, making them very easy to get started with. What’s more, both solutions follow ingestion option two above, and capture data live from dbt’s in-memory Jinja execution context during the run itself. Unfortunately, neither is super lightweight but they both offer additional capabilities to monitoring the health and performance of your dbt pipelines.

Conclusion

I get asked often about dbt metadata, and how to ingest it into Snowflake for operational use cases. Hopefully this blog post has given you a solid working knowledge of dbt metadata — what’s available and how to work with it.

In the next two blog posts we’re going to build on this baseline knowledge, towards the goal of enabling advanced dbt use cases such as state-aware deployment and orchestrations with Snowflake dbt Projects!

So stay tuned for the second blog post in the series, on dbt Metadata and Snowflake dbt Projects.

Blog posts in this series:

1. dbt Metadata Essentials — Part 1 (this blog post)
2. dbt Metadata and Snowflake dbt Projects — Part 2
3. dbt Metadata and State-Aware Operations — Part 3