Organisations are moving rapidly toward agentic workflows, enabling business users to query data in natural language without requiring deep technical expertise. As these agents are now directly involved in critical business decisions, updating them requires strict engineering rigour. Managing agent versions, automated evaluation, structured approvals, and rapid rollbacks are non-negotiable. This article explores a practical approach to managing Snowflake Cortex Agent versioning and multi-environment deployment.

Most data teams already have a release process for their data. This article uses one such setup as an example: dbt transforms and loads the silver and gold tables, and builds the views and semantic views on top of them. These dbt models are built in a development environment, validated in test and promoted to production, often with a Git branch for each environment. Cortex Agents add one more layer to this process. An agent has its own configuration, made up of instructions, models, and tools, and this configuration also needs to move from one environment to the next. Before building the pipeline, it is critical to decide whether a single agent should serve all environments, or whether each environment should have its own agent.

One agent for all environments, or one agent per environment?

Snowflake’s versioning model is built around a single agent. Changes are made in the live version, committed, tested through an alias and, once approved, set as the default version for CoWork users. Rolling back means moving the alias and the default version back to the previous version.

When a single agent is a good fit

A single agent fits when the versions at each stage use the same tools and data. Changes are then limited to the agent configuration, such as instructions, sample questions or model, and promoting a version releases exactly what was tested.

Consider a FinOps assistant built by the platform team. It answers questions such as “Which warehouses used the most credits last month?” using a single semantic view over Snowflake’s ACCOUNT_USAGE views. Snowflake provides these views, and each account has only one copy of them. Because the underlying data and tools exist only once, only the agent configuration changes between releases. The team creates a new version, tests it using the staging alias, and then moves the production alias to it.

Separate databases per environment: comparing the two options

Many organisations keep separate databases for each environment, so the agent’s semantic views and search services also have Dev, Test and Prod copies.

A single agent can still serve all three, with a separate version for each environment. The alternative is to create one agent per environment, for example `SALES_AGENT` in each of `SALES_DEV`, `SALES_TEST` and `SALES_PROD`, all deployed from the same Git template. Here is how the two options compare across promotion, version history, the draft, the default version and access:

Both options work. A single agent across environments is technically possible, but on every point above, separate agents keep each environment’s history, draft, default version and access independent, in the same way the databases already are. Each agent still uses Snowflake’s alias pattern for testing, release and rollback.

Tool changes vs agent changes: when does an agent need a new version?

An agent refers to its tools, such as semantic views and search services, by their fully qualified names in tool_resources. If a semantic view or search service is recreated under the same name, for example when dbt rebuilds it, the agent picks up the new definition without needing to ALTER the agent. Users see the change immediately, so the agent should be re-evaluated after such changes. Agent evaluation is covered later in this article.

A new agent version is needed when the agent specification itself changes, for example when a tool is renamed, a tool is added or removed, a tool’s configuration changes (such as filters or the warehouse), or the instructions or model change. Named versions cannot be changed once committed, so each of these changes is committed as a new version.

Keeping grants intact

When dbt or a pipeline recreates an object with `CREATE OR REPLACE`, the object’s grants need attention. Adding `COPY GRANTS` keeps the explicit privileges that were granted on the original object, except OWNERSHIP. Future grants are not copied. Without `COPY GRANTS`, the explicit grants are lost, and roles that could use the object before may lose access to it. Agents and semantic views both support `COPY GRANTS`.

Version changes made with `ALTER AGENT`, such as adding a version, moving an alias or setting the default version, are different. They change the existing agent in place, so the agent and its grants remain.

From pull request to production: the release pipeline

With one agent per environment, every change follows the same path (Dev -> Test -> Prod) including small updates to instructions or sample questions. No environment should be skipped, for three reasons:

• Small changes can still change behaviour. A one-line edit to the instructions or a new sample question can change how the agent picks tools or phrases answers. Its effect is only known once the change has been tested.
• Environments stay in step. If a change goes straight to Prod, Dev and Test no longer match it. The next change is then built and tested against an outdated version, and the direct change has to be back-merged into the lower branches. If it is not, the next release from Dev overwrites it with the outdated version.
• Every release has the same record. When each change passes the same reviews, evaluations and approvals, it is always clear what was tested and approved before it reached users.

Change tiers: the same path, different checks

Not every change carries the same risk, so the checks can differ even though the release path does not. Grouping changes into tiers makes this explicit:

Each tier’s checks and approvals, such as code reviews, user testing and production sign-off, should follow your organisation’s existing release process. When a change spans several tiers, it follows the strictest one.

Pipeline steps

Here are the recommended steps for deploying data and agent changes. The same steps run in each environment, against that environment’s own objects.

1. Review (PR + tier check): A pull request is raised, and an automated check classifies the change into one of the tiers above, based on what the PR contains. The tier decides which reviews and approvals the change needs.

2. Plan & validate (detect changes): The pipeline detects what has changed: the dbt models, the agent specification, or both. This decides which of the next steps run. It then validates the changed files before anything is deployed, for example by compiling the dbt project and checking that the agent specification is valid.

3. Build data (dbt, if changed): If the dbt models changed, dbt builds the tables, views and semantic views. If only the agent changed, this step is skipped.

4. Deploy agent (new version): If the agent specification changed, a new agent version is created from the Git specification, using that environment’s object names.

5. Evaluate (score + compare): The new version is scored against the agreed question set and compared with the current version. For a data-only change, the current version is evaluated against the new data instead.

6. Promote (goes live): If the evaluation passes and the approvals for the tier are in place, the new version is made live by moving the production alias and the default version to it.

Deploying data and the agent

dbt builds the tables, views, semantic views and other objects the agent depends on. The agent is then deployed from its specification in Git as a new version.

Before a new version is created in an agent that is serving users, pin `DEFAULT_VERSION` to the approved version. Otherwise the new commit becomes the default straight away, before it has been evaluated.

A change that touches only data needs care. dbt rebuilds the objects, the agent version stays the same, and the current version is re-evaluated against the new data. Because the new data is already live when the evaluation runs, this evaluation is a check, not a gate: it can find a problem, but it cannot stop the change from reaching users.

Evaluate, promote and roll back

Cortex Agent Evaluations score an agent against a set of questions. Each run calls the agent again for every question and scores the response.

To make results comparable from one run to the next, evaluate a committed version, not the editable draft, and pin the metric versions used to score it.

Agree on a question set with the business team for each agent, with the expected answers. Then agree a minimum score for each agent. Agent responses vary between runs, so run the same questions several times first to learn the normal range of scores, and set the threshold based on that range rather than a single run. Naming the orchestration model, instead of using auto, keeps model changes from being mistaken for normal variation.

If the new version scores below the threshold, or noticeably below the current version, the pipeline stops and the change is not promoted; users stay on the current version, so nothing needs to be rolled back. For a data-only change, a failed check means the data change is fixed or reverted, because it is already live. If it passes and the approvals for the tier are in place, the `production` alias and the default version are moved to the new version, so CoWork users get it straight away, and the previous version is kept.

If a problem appears after promotion, the alias and default are pointed back to the previous version, which restores the agent configuration, but not the data or tools it uses.

It is recommended that any edits made in the Snowsight UI are copied back into Git through a pull request, so Git stays the source of truth.

Key takeaways

• When tools live in separate databases per environment, use one agent per environment; a single agent fits when tools and data exist only once.
• Use committed versions for production and pin agent and metric versions for comparable evaluations.
• Calibrate evaluation gates with repeated runs and business-specific acceptance criteria.
• Validate tool changes and preserve intended access when replacing objects.
• Follow the same Dev -> Test -> Prod path for every change, with checks matched to the change tier.

The architecture and release-process recommendations in this article reflect my own views, not an official Snowflake reference architecture. This includes when to use one agent or separate agents, the combined dbt-agent pipeline, change tiers, and thresholds and approvals. Adapt the approach to your team’s requirements.

Cortex Agent Versioning and Multi-Environment Deployment was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Enterprise ML architectures carry history. Many were built when the data platform handled storage, transformation and analytics, while a separate ML platform supplied notebooks, training infrastructure, registries, deployment pipelines, online endpoints, monitoring and specialist compute. Those choices created skills, controls and production services that may have improved over years. Snowflake now provides capabilities across more stages of the ML lifecycle. These capabilities now include feature management, development and job execution, model governance, warehouse and container-based inference, observability, explainability and lineage. The platform can also work with models trained elsewhere and with inference performed outside Snowflake. These broader capabilities merit reassessment, not automatic migration.

A platform gaining a capability is not, by itself, an architecture justification for moving a workload. Placement should follow workload evidence: data movement, execution pattern, latency, feature reuse, operating maturity, prediction consumption, governance and total cost. Here is a set of seven tests to structure that discussion and decision making.

Test 1: Does data movement materially affect the workload?

“The data is already in Snowflake” is not a sufficient reason to run a machine learning workload there. Locality matters when the volume, frequency and operational consequences of crossing the platform boundary are substantial.

As an example, consider a churn model that scores 50 million customer records each night. Source data and features reside in Snowflake, and the probabilities return there for campaign segmentation. External scoring creates a round trip; extract a large feature population, transfer it to another service, score it, move the predictions back, reconcile the result and recover from failures across two platforms. Running inference closer to the governed data may remove meaningful work.

Contrast that with a mature low-latency endpoint receiving a vector of 20 precomputed features from an operational application. The payload is small, the prediction is consumed immediately by the application, and the service already meets latency, scaling, monitoring and release requirements. Moving it merely because another platform can host inference may replace a well-run boundary without solving the material problem.

Measure the boundary rather than treating locality as an architectural virtue.

Test 2: What are we paying to simplify?

Compute rates are only one part of the comparison. A credible ML total-cost view includes compute, data movement, duplicated feature logic, integration, platform operations, governance, skills and change cost.

Direct infrastructure may be economical when utilisation is high, commercial commitments are favourable and the operating team is effective. Managed execution may cost more per unit yet reduce orchestration, staging, reconciliation and support effort. Neither result can be assumed from a price list.

For container workloads, Snowpark Container Services uses managed compute pools, while storage and data transfer remain separate cost considerations. Warehouse inference has a different operating and pricing profile again. Each option must be tested against the workload’s region, service objective, concurrency, scaling pattern and utilisation.

Change cost deserves equal weight. Rebuilding a mature service can require retraining teams, recreating security and approval controls, altering CI/CD and on-call ownership, and accepting transition risk. Conversely, a batch pipeline may justify change if it removes repeated extraction, feature duplication and cross-platform reconciliation. Benchmark the workload, not an abstract platform configuration.

Test 3: Is the workload batch, online or specialised?

Inference covers operating patterns with different architecture needs. A nightly propensity run, a millisecond-sensitive fraud decision and a GPU-intensive vision model should not share a placement rule.

Snowflake is a candidate for large batch scoring over Snowflake tables, forecasting across many entities, anomaly detection embedded in data pipelines and model inference executed from Snowflake compute. Warehouse inference can reduce movement when the scoring population and downstream consumers already sit in Snowflake. Models do not need to be trained there first; externally trained models can be registered and invoked where supported.

Online and specialised workloads need harder evidence. Latency, throughput, concurrency, availability, autoscaling, network path, regional deployment, accelerator needs, custom runtimes and burst behaviour can outweigh locality. Snowpark Container Services expands the available execution choices, including specialised compute, but there is a need to establish workload suitability and price-performance.

Existing production services face a higher hurdle for change. Proven latency, effective monitoring, mature deployment automation, security controls and clear operational ownership are assets. A migration must produce measurable value beyond architectural symmetry.

The result may differ by execution mode. One model can use warehouse inference for scheduled portfolio scoring while retaining a real-time endpoint for transactional decisions. Shared model logic does not require a single serving pattern.

Test 4: Where should features live?

Feature placement can be more consequential than model placement. Recreating features independently produces inconsistent definitions, point-in-time leakage, stale values, training-serving skew and unclear lineage.

A governed feature layer in Snowflake can be valuable when the underlying data already resides there and several teams reuse the same transformations. Feature Store can centralise feature definitions and governed access while supporting configured refresh for feature data. These capabilities change the decision only if they reduce duplicate engineering and establish reliable ownership.

Centralising features does not require centralising the whole lifecycle. A sound hybrid design can keep governed data and reusable features in Snowflake, use a specialised training environment, deploy to the serving environment that meets operational needs, and return predictions or ground truth to Snowflake for governed consumption and analysis.

Online features require separate scrutiny. Their freshness, access path and latency may call for a dedicated serving layer even when offline definitions originate in Snowflake. The architecture should preserve semantic consistency without forcing every read through the same runtime. Consolidate feature logic when it creates reuse, point-in-time correctness and accountable lineage. Do not build a feature store merely because the platform offers one.

Test 5: Where should training, governance, execution and observation occur?

Training, governance, execution and observation are separate placement decisions. Treating them as one platform choice hides the requirements that differ across the lifecycle.

Training may depend on experiment velocity, distributed compute, accelerators, open-source dependencies and existing team practice. Governance depends on ownership, versioning, approval, access control, lineage and promotion between environments. Model Registry can govern supported models, including models trained externally, without requiring their development history to move.

Execution should follow the serving mode. Warehouse inference and Snowpark Container Services provide different execution options, but both remain workload-specific choices.

Observation depends on where predictions and outcomes can be joined reliably. For models monitored through Snowflake ML Observability, stored predictions, features and ground truth can be used to assess model performance, drift and prediction volume. That supports a useful hybrid pattern: an external service produces online predictions, Snowflake stores those predictions, ground truth arrives later, and monitoring runs against governed data. Explainability and lineage are useful where they support review, investigation or regulation; they should not be added as generic platform features.

As an example, the lifecycle can be deliberately split: specialised training outside Snowflake, governance in Model Registry, batch inference in a warehouse, online serving externally, and observation against data in Snowflake.

Test 6: Where are predictions consumed?

Prediction destination can matter as much as training-data location. If a batch score returns immediately to a Snowflake customer table for segmentation, analytics or governed activation, an external inference boundary should demonstrate the value it adds. Otherwise it may create movement and orchestration without changing the business outcome.

The answer is different when an operational application sends a small request to a real-time endpoint and acts on the response immediately. If Snowflake is neither the feature-serving path nor the prediction consumer, relocating inference may add a network hop or new dependency with little offsetting benefit.

Consumption also determines the recovery model. Batch predictions written into governed tables can be reconciled, rerun and audited with data pipelines. Transactional predictions may need synchronous fallbacks, strict availability and application-level telemetry. Those are different operating contracts even when the model artefact is related.

A single model can therefore support more than one execution path. A risk model might score a full portfolio overnight near the data while an external service handles new transactions. Consistent model and feature governance matters; identical infrastructure does not. Always follow the prediction into the business process. Its destination reveals the meaningful latency, security, orchestration, reconciliation and monitoring requirements.

Test 7: Does an agentic workflow change the placement decision?

Predictive models estimate probabilities, forecasts, rankings or expected outcomes from historical patterns. CoCo can improve development work such as exploration, feature engineering, training, evaluation, debugging and deployment preparation. CoCo capabilities are evolving, so its influence must be carefully considered after validating its availability for the target environment.

None of this dictates inference placement. An agent operating near Snowflake data may call a model served elsewhere because that endpoint still provides the right latency, runtime or operating model. Conversely, a Snowflake-resident predictive signal may reduce movement when the agent and its governed context already consume data there.

The placement question remains unchanged: which boundary best satisfies the predictive workload and the business action? Agentic workflows increase the value of reliable, governed signals; they do not remove the need to benchmark how those signals are produced and served.

Turning the tests into an architecture decision

Evidence should include observed workload behaviour rather than stakeholder preference: representative volumes, latency distributions, utilisation, failure and recovery experience, operational effort, control requirements and commercial assumptions. Where the evidence is incomplete, the recommendation should remain conditional rather than being forced into a platform decision.

The seven tests structure a conversation; they do not produce a generic suitability percentage. Constraints are asymmetric. A hard latency objective, regulatory restriction, specialised runtime or mature production service cannot sensibly be averaged away by several weaker benefits.

Four outcomes provide a useful decision vocabulary:

• Keep — Retain the architecture when the existing boundary earns its cost through proven service levels, mature operations, suitable specialist infrastructure, effective governance or predominantly external consumption. A credible review must be able to recommend no migration.
• Integrate — Improve selected interfaces without replacing working capabilities. Govern features in Snowflake while retaining specialised training; keep an online service while returning predictions and ground truth for observation; or use warehouse inference for batch scoring while preserving a separate transactional path.
• Consolidate — Move the lifecycle components that show measurable benefit. Strong candidates combine a large Snowflake-resident scoring population, repeated boundary movement, batch execution, Snowflake-derived features, Snowflake-centric consumption and manageable change cost.
• New Build — For a new workload centred on governed Snowflake data, reusable features, batch execution and Snowflake consumption, evaluate a Snowflake-based design before adding another platform boundary. Another platform may still win, but the boundary should have a demonstrated purpose.

Snowflake’s expanding ML capabilities create more placement choices, not a reason to consolidate by default. Existing investments, workload characteristics and operating constraints still matter, and the right architecture may be consolidated, external or deliberately hybrid. The decision should follow evidence from the workload tests and the value created by changing the boundary, not the availability of a platform capability. The responsibility is not to choose a platform first. It is to make every architecture boundary earn its cost.

Enterprise ML Architecture: Making Every Boundary Earn Its Place was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Every few weeks someone asks me the same thing: “We want a Cortex Agent on top of our data. How should we model it?” The honest answer is that most of the questions behind that sentence are not about the agent. They are modelling questions we should have answered years ago, and an agent now exposes them in public.

A business analyst carries a lot of tacit knowledge: which date really means “sale”, which orders are cancelled, what “revenue” means for Finance versus Sales. An agent has none of that. It reads your schema, infers, and writes SQL. Every ambiguity becomes an inference, and every inference is a chance for a wrong join, filter or aggregation. In this article we will walk through the six questions I ask teams before they start modelling: grain, shape, semantic views, business meaning, performance and evaluation. We will close with a checklist you can take to your next design session.

Index

• 1. Why agents change the rules
• 2. Question 1: What is the grain, and does every column mean one thing?
• 3. Question 2: Star, snowflake or wide table, and how many joins are too many?
• 4. Question 3: How big should a Semantic View be?
• 5. Question 4: Who owns the meaning of a metric?
• 6. Question 5: Does physical performance still matter?
• 7. Question 6: How will we know it works?
• The Big Picture
• Conclusions
• References
• About Me

1. Why agents change the rules

The first thing I explain is that an agent solution is not one model; it is a stack of layers, each with a different job. Confusing them is the root of most disappointing proofs of concept.

• Curated physical model: tables, views, facts, dimensions, types, keys and quality tests. It represents the data faithfully and consistently.
• Semantic View: the executable semantic layer. It describes how to query a domain and how to calculate its metrics: logical tables, relationships, dimensions, facts, metrics, filters and descriptions.
• Business Ontology: the governed meaning of concepts (terms, metrics, entities) across models.
• Cortex Sense: AI-derived context, useful where data is not fully modelled yet.
• Cortex Agent: the orchestrator. It uses Cortex Analyst for structured data, Cortex Search for unstructured content and any other configured tools, and combines the results.

Key idea: none of these layers fixes physically incorrect data. Answer quality comes from four things together: correct grain and relationships, explicit semantic definitions, governed business context and systematic testing. A good design reduces the number of inferences the language model has to make. That is the guiding principle for everything that follows.

The five layers: each one has a different responsibility, and none replaces the layer below.

[↑Up]

2. Question 1: What is the grain, and does every column mean one thing?

The grain

Every fact table needs a grain written as a verifiable sentence: “one row per order line”, “one row per product, warehouse and day”. If the team cannot write it, the model is not ready.

• Do not mix grains: avoid headers and lines, detail and subtotals, or transactions and snapshots in the same consumption table.
• Separate current from history: with SCD Type 2, expose the current state and the full history through explicit views. Otherwise a query will happily count several versions of the same customer.
• Test uniqueness: on standard Snowflake tables, PRIMARY KEY and UNIQUE constraints are not enforced. Quality tests are still your responsibility.

Columns, additivity and dates

• Business names: net_amount_eur and invoice_date beat amt2 and dt.
• One meaning per column: state the unit and the rules (“net amount after discounts, in EUR”). Say what a NULL means: unknown, not applicable or anonymised.
• Codes with descriptions: expose status_code next to status_description.
• Flatten semi-structured data in the consumption layer: keep VARIANT in raw, expose typed columns for what matters.
• Hide technical columns: load_id, hash_row and etl_batch_id have no analytical value and only add noise.
• Know the additivity: a balance or a stock level is semi-additive and needs “last value of the period”, not a SUM. Distinct counts and historical metrics need specific logic too.
• Name every date: order, invoice, shipment, delivery and posting dates are different things. Put fiscal periods, weeks and holidays in a calendar dimension or an explicit semantic definition.

People always ask how many columns a table should have. There is no universal number. Relevance, clear descriptions and the absence of redundant or confusable attributes matter far more than any fixed limit.

[↑Up]

3. Question 2: Star, snowflake or wide table, and how many joins are too many?

Choosing the pattern

• Star schema: my default starting point. Clear paths and joins that are easy to describe.
• Conformed dimensions: the right way to compare several processes (sales and returns, say) without joining transactional facts directly.
• Wide table: fine for a bounded case, provided there is a single clear grain and you control duplication.
• Snowflake / 3NF / Data Vault: more joins and more possible paths. Keep it underneath, and expose a simplified logical layer on top for better accuracy and performance responses.

Relationships that leave no room for doubt

• Declare them explicitly: never rely on similar column names. Cortex Agents will not join tables unless the relationship is defined.
• Watch for fan-out: especially between tables of different grain.
• Many-to-many: resolve it with an explicit bridge table and document how measures are distributed. Snowflake’s guidance is to use a shared bridge table rather than a direct M:N relationship.
• Role-playing dimensions: order date and ship date are two roles of the same calendar. Model each role separately.
• Avoid fact-to-fact joins: unless grain and cardinality are fully controlled. Shared dimensions and purpose-built metrics are safer.

So, how many joins are too many?

There is no magic number. What matters is cardinality, referential integrity, alternative paths, optional dimensions and grain compatibility. One simple join with the wrong relationship is far more dangerous than five well-defined ones. The goal is not to minimise joins at any price, but to offer an unambiguous query path that supports the metrics you want to compute. Weigh simplicity against the functionality your data product needs.

The same logic applies to aggregates. They can cut latency and cost, and can encapsulate complex calculations such as balance snapshots. Give them an explicit grain in the name (sales_monthly_by_region), validate additivity before reusing them, and keep physical optimisation below a stable Semantic View so the agent never has to choose between detail and aggregate tables.

[↑Up]

4. Question 3: How big should a Semantic View be?

Smaller than you think. Snowflake’s own modelling guidance suggests starting with a limited scope and a simple star schema, with roughly 5 to 10 tables for an initial proof of concept. That is a starting reference, not a ceiling. It also recommends keeping the semantic context under about 100,000 tokens, because oversized views add latency and can lead to context being pruned.

Scope and organisation

• One domain or use case per view: sales performance, finance, inventory, customer service. A single view for the whole company adds ambiguity and latency.
• Curated objects only: do not point the agent at raw or staging schemas.
• Cross-domain questions: build an integrated layer or a dedicated view for them.

What gives accuracy

• Descriptions: they have the biggest impact. Describe business meaning and important restrictions for every table, column, fact and metric.
• Synonyms: use sparingly, for real corporate vocabulary and tested alternative names.
• Metrics and filters: fix the logic once. “Net sales” must state whether cancelled orders, returns and taxes are in or out, and be validated. A filter such as “valid order” must not depend on the agent guessing from a previous question.
• Verified Queries: reviewed question and SQL pairs for frequent, ambiguous or complex questions. They help consistency, but they do not replace a correct model or continuous evaluation.

Creating and managing

Semantic View Autopilot is a great accelerator for the first draft, from tables, sample queries or imported BI models, but always review relationships, metrics and descriptions. For anything that goes to production, I manage Semantic Views as code with SQL DDL or YAML, using native Semantic View objects rather than the legacy stage-based YAML models. A minimal sketch (Illustrative only):

CREATE OR REPLACE SEMANTIC VIEW sales_sv
  TABLES (
    order_lines AS analytics.curated.fct_order_line
      PRIMARY KEY (order_line_id)
      COMMENT = 'One row per order line',
    customers AS analytics.curated.dim_customer
      PRIMARY KEY (customer_id)
      COMMENT = 'One row per current customer'
  )
  RELATIONSHIPS (
    lines_to_customers AS order_lines (customer_id) REFERENCES customers
  )
  FACTS (
    order_lines.valid_net_amount_eur AS IFF(COALESCE(order_status, '') <> 'CANCELLED', net_amount_eur, 0)
      COMMENT = 'Net amount after discounts, EUR, cancelled orders excluded'
  )
  DIMENSIONS (
    customers.segment AS segment_name
      COMMENT = 'Commercial segment of the customer',
    order_lines.order_date AS order_date
      COMMENT = 'Date the order was placed (not invoice or ship date)'
  )
  METRICS (
    order_lines.net_sales AS SUM(order_lines.valid_net_amount_eur)
      COMMENT = 'Net sales EUR: after discounts, excluding cancelled orders'
  );

[↑Up]

5. Question 4: Who owns the meaning of a metric?

“Revenue” can mean invoiced sales for Sales, sales net of discounts for Commercial, or recognised revenue under an accounting rule for Finance. All three can be right in their own domain. The problem starts when a question does not say which one applies, and the agent silently picks one.

Business Ontology

Business Ontology is a governed business-meaning layer within Snowflake Horizon Catalog. Data owners declare an approved definition for a term, metric or entity, model its relationships, and link it to the tables, views, columns or Semantic Views that implement it.

• Term: a business concept in plain language, such as “Net sale”.
• Metric: a concept defined by a formula or KPI, such as “Net Revenue”.
• Entity and Relationship: business objects and typed links, such as “an order contains order lines”.
• Representation: the link between the concept and an asset that implements it, for example a metric in a Semantic View.

The value is separating the canonical business concept from any single technical implementation. Net Revenue can have representations in several assets, with one approved definition as the common reference.

Availability note: at the time of writing (October 2026), Business Ontology, its Cortex Agents integration and Cortex Sense are in Private Preview for selected accounts. Check availability for your account before designing around them.

How the layers differ

• Semantic View: governs how a domain is queried and how metrics are calculated. It is not an enterprise-wide glossary.
• Business Ontology: governs approved definitions, relationships and links to assets. It does not execute a metric’s formula.
• Cortex Sense: brings AI-inferred context. It is not the same as a definition approved by a business owner.

The critical limit: documented formula versus executed formula

In the documented integration, the formula stored in the ontology is semantic documentation. Cortex Analyst computes the value using the expression in the Semantic View. If the two drift apart, the agent explains one thing and calculates another. Keep them aligned and test them.

Governance practicalities

• Lifecycle: nodes are Draft, Approved or Deprecated. Only approved definitions are served as canonical.
• Separate permissions: ontology domain access and privileges on Semantic Views and tables are different controls. Test them per role.
• Links are not revalidated: if a target asset is dropped or renamed, validate the representations in your deployment process.
• Agent tool descriptions: if the agent covers several domains, configure one Analyst tool per Semantic View, with a description that clearly states what data it covers, and test tool selection.
• Ambiguity: if two definitions are equally plausible, the agent should surface the difference and ask, not guess.

[↑Up]

6. Question 5: Does physical performance still matter?

Absolutely, but do not confuse it with correctness. A query can be semantically right and expensive, and a fast query can be wrong because the model duplicates rows or applies the wrong filter. Both need to be designed.

• Types and predicates: keep types consistent with the content and avoid casts on filtered columns.
• Pruning and clustering: justify clustering with real access patterns and volume. “Always cluster by date” is not a rule.
• Pre-aggregation: Dynamic Tables, materialised views or other structures, when they remove repeated work and the refresh cost is reasonable.
• Concurrency and latency: size warehouses and evaluate execution options against your SLA and query pattern.
• High-cardinality text: Cortex Search helps to find entities by names, aliases or fuzzy text. It does not replace relational keys or metric logic.
• Measure: profile the generated SQL, bytes scanned, execution time and warehouse behaviour under realistic load.

Validate on representative business questions, not on a synthetic query. And measure the whole chain: tool selection, SQL generation, execution and answer composition.

[↑Up]

7. Question 6: How will we know it works?

If you cannot answer this, you are not ready to publish the agent. Build an evaluation set from real business questions, recording for each one the question, the domain, the applicable metric definition, the expected result and, where relevant, the validated SQL. Snowflake’s guidance suggests starting with around ten representative benchmark questions and using the native Cortex Analyst evaluations to measure SQL accuracy. These are the categories I always cover:

• Simple: filters, groupings and aggregations on one table.
• Relational: several joins, composite keys, date roles and shared dimensions.
• Double-counting risk: facts of different grain, N:M, snapshots and semi-additive measures.
• Semantically ambiguous: alternative names, homonymous metrics and per-domain rules.
• Security: authorised and unauthorised users, domain access and restricted objects.
• No definition: the agent must recognise it lacks context and must not invent a formula.

Re-run the evaluation whenever the Semantic View, ontology, instructions, tools, permissions or agent configuration change.

Who owns what

• Data engineering / architecture: grain, quality, keys, relationships, physical performance and integrity tests.
• Technical semantic owner: Semantic Views, executable metrics, filters, Verified Queries and deployments.
• Data steward / business: canonical definitions, synonyms, domains and approval of ontology concepts.
• AI / application team: agent tools, orchestration instructions, tool-selection tests and behaviour under ambiguity.
• Security and governance: role permissions, access reviews and tests of allowed and restricted requests.

[↑Up]

The Big Picture: From Tables to Trusted Answers

Each layer reduces ambiguity for the one above it, and evaluation closes the loop.

[↑Up]

Conclusions

Modelling for AI agents is not a new discipline, it is the old one with the tolerance for ambiguity removed. The physical model must be correct and at a clear grain. The Semantic View must make the query path and the metric logic explicit. The business meaning must be owned by someone who can approve it. And everything must be tested, repeatedly, with questions your users actually ask.

Key takeaway: before you open the Semantic View editor (Semantic Studio), answer the six questions: what is the grain, which shape and which joins, how big is each view, who owns the meaning, how will it perform, and how will we know it works. If you cannot answer one of them, that is where the next sprint should go. A quick checklist to take with you:

• ☐ The grain of every fact table is written down and validated with data.
• ☐ Keys, cardinalities, fan-out and double-counting have specific tests.
• ☐ Visible tables and columns are relevant and have business descriptions.
• ☐ Metrics have an executable, validated definition, and relationships and date roles are explicit.
• ☐ Cross-domain concepts have a canonical definition and an owner, and documented and executed formulas are aligned.
• ☐ Agent tools describe clearly what data they cover, and ambiguity is surfaced to the user.
• ☐ Permissions on ontology, Semantic Views and tables are tested per role.
• ☐ An evaluation set measures accuracy, latency, cost and regressions after every change.

One last word of caution: some of the capabilities discussed here are still in preview at the time this article was written and could evolve quickly. Treat these notes as design principles, re-check availability for your account, and let your domain, workload and enabled features drive the final decisions.

[↑Up]

References

This list contains most of the key content in the documentation, that I make reference in this article and it could help you:

Business Ontology (Preview) — Concepts, relationships, representations, governance, and state of functionality.

Use Business Ontology with Cortex Agents (Preview) — Agent integration, definition query, Cortex Analyst, permissions, and limitations.

Cortex Sense (Private Preview) — Working model of your business from metadata your account already produces.

Best practices for modeling semantic views — Scope, descriptions, synonyms, relationships, metrics, filters, and tests.

Cortex Agents — Orchestration of tools for structured and unstructured data.

Overview of Semantic Views — Introduction to Semantic Views.

CREATE SEMANTIC VIEW — A SQL reference for creating Semantic Views.

Cortex Analyst evaluations — Cortex Analyst quality and results evaluation.

Semantic View Autopilot — Assisted creation of Semantic Views.

Creating semantic views — Authoring options using Autopilot, SQL, YAML, or existing models.

YAML specification for semantic views — YAML specification and differences from legacy semantic models.

Using SQL commands to create and manage semantic views — Managing Semantic Views with DDL.

Get started with Cortex Agents — Initial guide to creating and configuring agents.

Routing mode for Cortex Analyst — Query generation strategy with Semantic Views and semantic guardrails.

Snowflake CoWork: Build agents — Configuration of agents, tools, instructions, and resources.

Snowflake CoWork: Troubleshooting — Common issues, response consistency, and optimization.

Cortex Analyst — Overview of natural language queries on structured data.

[↑Up]

About Me

As a Snowflake Principal Architect & SME on different Data Technologies at SDG Group and 20+ years of experience in several Data Adventures, I lead and advise teams to get the best approaches and apply best practices.

I am a Snowflake Data Superhero since 2025, Snowflake Certified on Advanced Architecture + GenAI Specialty, Snowflake Spotlight Squad Team Member 2024 and Snowflake Barcelona User Group Lead Chapter. As a Data Vault Certified Practitioner, I have been leading Data Vault architectures using Metadata Driven methodologies.

If you want to know more in detail about the aspects seen here, or other ones, you can follow me here on Medium || LinkedIn.

[↑Up]

Before You Model a Single Table: Data Modelling Questions to Ask for AI Agents in Snowflake was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

On Monday, someone improves the summary prompt. It’s a good change: the output is shorter, with no chatty preamble.

On Tuesday, the backfill starts. It re-summarizes every ticket since spring, at full token price.

On Wednesday, it’s still running. Last quarter’s report no longer matches the dashboard. The backfill reworded some old summaries, even at temperature 0.

Nobody touched those tickets. The pipeline rewrote them anyway.

On Thursday, two questions arrive from outside the data team:

“Can we improve a prompt without paying for every ticket again?” No. With this pipeline, every backfill pays for your whole history again, and history only grows.

“Can we improve a prompt without changing last quarter’s numbers?” No. Every prompt change reopens a report that you already shipped.

This post builds a pipeline where both answers are yes.

TL;DR: Put AI_CLASSIFY, AI_FILTER and AI_COMPLETE in the SELECT of a Snowflake dynamic Iceberg table. The AI step then becomes a layer of your lakehouse that any Iceberg engine can read, not a job beside it. Snowflake calls the model only for new and changed rows. A frozen region keeps the old outputs when the prompt changes. You'll build a ticket-enrichment pipeline from two CREATE statements, with no scheduler, history table or backfill script.

Customers already use Dynamic Iceberg Tables to turn changing source data into fresh, business-ready datasets in open formats, like current-state network inventories, clinical data products, and regional delivery feeds, without building custom merge pipelines. With externally managed dynamic Iceberg tables, now in Private Preview, Snowflake runs those same declarative, target-lag refreshes and publishes the results straight into your existing external Iceberg catalog, so you keep your catalog and still get Snowflake-managed pipelines

You’ll need:

• An account where Snowflake storage for Iceberg is available (I used AWS), or an external volume of your own
• A role that can create a database and call AI Functions (by default, every role can call them, through PUBLIC)
• An XS warehouse (in this post, its name is transform_wh)
• Python with pyiceberg and pyarrow, for Step 6

How to read this post: 6 short steps come first, each with tested SQL. The known problems and the two answers follow. For safe prompt changes, go to Step 4.

From a scheduled job to a lakehouse layer

Usually, an Airflow job or a dbt job runs each hour. Each run has these steps:

1. It finds the new rows.
2. It calls the model.
3. It upserts the results into a history table, so it never pays twice.
4. It retries the NULL results.
5. It checks the load.

When the prompt changes, someone writes a backfill. The job also sits outside the lakehouse. Its logic lives in a scheduler: which rows are new, which rows to retry and when to backfill.

In this post, an Iceberg table holds the raw tickets. One dynamic Iceberg table runs the AI Functions in its SELECT. A second one filters the safety cases.

Snowflake finds the rows that changed and refreshes the tables to meet a TARGET_LAG. A TARGET_LAG is a freshness goal, not a schedule (best practices). Both outputs are standard Iceberg tables, so the AI step becomes one more layer of the lakehouse.

Step 1: Load the tickets into an Iceberg table

EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED' keeps the files in Snowflake storage, so you need no bucket and no IAM role. CHANGE_TRACKING = TRUE lets dynamic tables read the table incrementally.

CREATE DATABASE demo_db;
CREATE SCHEMA demo_db.demo;
USE SCHEMA demo_db.demo;

CREATE ICEBERG TABLE support_tickets (
  ticket_id   INT,
  created_at  TIMESTAMP_NTZ(6),
  channel     STRING,
  body        STRING
)
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
  CHANGE_TRACKING = TRUE;

INSERT INTO support_tickets VALUES
  (1,  '2026-09-15 08:12:00', 'email',
    'I was charged twice for order 4471. '
    || 'Please refund the duplicate charge.'),
  (12, '2026-10-03 14:45:00', 'chat',
    'My order is three days late and tracking has not updated.'),
  (18, '2026-10-11 13:00:00', 'phone',
    'The charger gets very hot and smells like burning plastic.'),
  -- ...16 more synthetic tickets, mid-September to mid-October
  (20, '2026-10-14 20:50:00', 'chat',
    'I love the new app update. The order history page is much faster.');

I loaded 20 synthetic tickets. Half of them are from before October 1, and this split is important in Step 4.

Step 2: Put the model in the SELECT

The statement below is the full enrichment job. It calls three AI Functions:

• AI_CLASSIFY chooses a topic.
• AI_FILTER answers a yes-or-no question about safety.
• AI_COMPLETE writes the summary.

CREATE DYNAMIC ICEBERG TABLE ticket_enriched
  TARGET_LAG = '1 hour'
  WAREHOUSE = transform_wh
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
  REFRESH_MODE = INCREMENTAL
AS
SELECT
  ticket_id,
  created_at,
  channel,
  body,
  AI_CLASSIFY(body, ['billing', 'delivery', 'account access',
                     'product defect', 'outage', 'other']
  ):labels[0]::STRING AS topic,
  AI_FILTER(PROMPT(
    'Does this support ticket describe a safety hazard to a person? {0}',
    body
  )) AS is_safety_issue,
  AI_COMPLETE(
    model => 'llama3.1-8b',
    prompt => CONCAT(
      'Summarize this support ticket in one sentence: ', body),
    model_parameters => {'temperature': 0, 'max_tokens': 60}
  )::STRING AS summary
FROM support_tickets;

Two details are important:

• Keep the AI Functions in the SELECT. An incremental refresh supports them only there (supported queries). With REFRESH_MODE = INCREMENTAL, any other place makes the CREATE statement fail. Without that mode, Snowflake uses a full refresh without a warning. Step 5 shows why the error is better.
• Cast the AI_COMPLETE output to STRING if needed. The function returned VARIANT. Iceberg v3 tables support this type, but the Iceberg v2 spec does not.

What you should see: SHOW DYNAMIC TABLES shows refresh_mode as INCREMENTAL. The first refresh enriched all 20 tickets in about 5 seconds and flagged exactly the three hazards. For a question about an invoice fee, it wrote the summary "There is no text to summarize." Outputs like this are the reason for the prompt change in Step 4.

Step 3: Pay only for what changed

The SQL below adds five tickets and refreshes the table. Then it reads the refresh history:

INSERT INTO support_tickets VALUES
  (21, '2026-10-15 08:05:00', 'email',
    'I returned the vacuum two weeks ago and still have no refund.'),
  -- ...three more
  (25, '2026-10-16 14:00:00', 'chat',
    'Can you combine my two orders into one shipment?');

ALTER DYNAMIC TABLE ticket_enriched REFRESH;  -- or wait for TARGET_LAG

SELECT refresh_action,
       statistics:numInsertedRows::INT AS inserted,
       statistics:numDeletedRows::INT  AS deleted
FROM TABLE(INFORMATION_SCHEMA.DYNAMIC_TABLE_REFRESH_HISTORY(
       NAME => 'DEMO_DB.DEMO.TICKET_ENRICHED'))
ORDER BY refresh_start_time DESC
LIMIT 1;

What you should see: an INCREMENTAL refresh that inserted 5 rows and deleted 0 rows. The query profile agrees: the model read 5 tickets, not 25. The first 20 rows kept exactly the same outputs.

Updates work the same way. I changed ticket 1 from “charged twice” to “charged three times.” Iceberg wrote a new version of the data file that holds ticket 1 and 19 other tickets, so 40 change records arrived. Snowflake combines these records into one net change for each row before the SELECT runs. 38 records cancel each other, and only the old and new versions of ticket 1 reach the AI Functions.

Step 4: Change the prompt and keep the old outputs

This is the Monday prompt change from the start of this post. First, freeze all rows from before October 1:

ALTER DYNAMIC TABLE ticket_enriched
  SET FROZEN WHERE (created_at < '2026-10-01');

SELECT COUNT_IF(METADATA$IS_FROZEN)     AS frozen_rows,
       COUNT_IF(NOT METADATA$IS_FROZEN) AS active_rows
FROM ticket_enriched;

My table now had 10 frozen rows and 16 active rows. The SET FROZEN statement did not start a refresh.

Next, change the prompt with CREATE OR ALTER. In the same statement, state the frozen region again:

CREATE OR ALTER DYNAMIC ICEBERG TABLE ticket_enriched
  TARGET_LAG = '1 hour'
  WAREHOUSE = transform_wh
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
  REFRESH_MODE = INCREMENTAL
  FROZEN WHERE (created_at < '2026-10-01')  -- keep this line every time
AS
SELECT
  ticket_id,
  created_at,
  channel,
  body,
  AI_CLASSIFY(body, ['billing', 'delivery', 'account access',
                     'product defect', 'outage', 'other']
  ):labels[0]::STRING AS topic,
  AI_FILTER(PROMPT(
    'Does this support ticket describe a safety hazard to a person? {0}',
    body
  )) AS is_safety_issue,
  AI_COMPLETE(
    model => 'llama3.1-8b',
    prompt => CONCAT(
      'In 12 words or fewer, say what the customer needs: ', body),
    model_parameters => {'temperature': 0, 'max_tokens': 60}
  )::STRING AS summary
FROM support_tickets;

What you should see: the next refresh is a REINITIALIZE. A REINITIALIZE recomputes only the active region (frozen regions). In my run, 15 of the 16 active summaries changed, but 0 of the 10 frozen summaries changed. For example, ticket 12 changed:

• Before: “The customer’s order is three days past the expected delivery date and the tracking information has not been updated.”
• After: “I need an update on the status of my delayed order.”

Ticket 2 is frozen, so its summary still starts with “Here is a summary of the support ticket in one sentence:”.

Include FROZEN WHERE in every CREATE OR ALTER. CREATE OR ALTER uses your statement as the complete target state (CREATE DYNAMIC ICEBERG TABLE). I ran the same change without the FROZEN line. The refresh reinitialized with the reason Frozen region changed or removed, and the model summarized all 26 rows again.

That run also showed the problem from the start of this post. With the same prompt, input and temperature 0, the model wrote different summaries for 2 of the 16 active rows. So do not run the model again on rows that are complete.

Step 5: Filter in a downstream table, not in WHERE

The obvious design for a safety queue puts AI_FILTER in the WHERE clause. With REFRESH_MODE = INCREMENTAL, the CREATE statement fails:

091912 (0A000): Query contains the function 'EXTERNAL_FUNC', but change
tracking is not supported on queries with non-deterministic functions.

With REFRESH_MODE = AUTO, the same statement succeeds. But Snowflake chooses FULL and does not tell you (supported queries). A full refresh sends every row to the model again, so each refresh repeats the backfill from the start of this post.

To fix this, filter on the flag that the upstream SELECT already computes:

CREATE DYNAMIC ICEBERG TABLE safety_queue
  TARGET_LAG = '1 hour'
  WAREHOUSE = transform_wh
  CATALOG = 'SNOWFLAKE'
  EXTERNAL_VOLUME = 'SNOWFLAKE_MANAGED'
  REFRESH_MODE = INCREMENTAL
AS
SELECT ticket_id, created_at, channel, topic, summary
FROM ticket_enriched
WHERE is_safety_issue;

What you should see: the refresh mode is INCREMENTAL. A new ticket about a kettle lid that burned a person's hand added 1 row to each table.

Step 6: Read the tables from outside Snowflake

Both tables are in the Snowflake catalog. Any engine that supports the Iceberg REST protocol can read them through Horizon Catalog. I used PyIceberg with a programmatic access token that I restricted to one role:

import os
from pyiceberg.catalog import load_catalog

uri = "https://<org>-<account>.snowflakecomputing.com/polaris/api/catalog"
props = {
    "type": "rest",
    "uri": uri,
    "oauth2-server-uri": f"{uri}/v1/oauth/tokens",
    "warehouse": "DEMO_DB",                   # database name, upper case
    "credential": os.environ["HORIZON_PAT"],  # never hard-code the token
    "scope": "session:role:<ROLE>",
    "header.X-Iceberg-Access-Delegation": "vended-credentials",
}
catalog = load_catalog("horizon", **props)
enriched = catalog.load_table("DEMO.TICKET_ENRICHED")
rows = enriched.scan(
    selected_fields=("TICKET_ID", "TOPIC", "IS_SAFETY_ISSUE", "SUMMARY")
).to_arrow()
print(rows.num_rows)

What you should see: PyIceberg returns the same 27 rows and summaries as Snowflake. SAFETY_QUEUE holds the five safety tickets.

The role needs USAGE on the database and the schema, and SELECT on the tables. Outside engines can read dynamic tables, but they cannot write to them.

Known problems

• A failed row does not cause an error, and it stays failed. I added a ticket with 672,036 characters, which is much larger than the 128K-token context window of llama3.1-8b. The refresh succeeded, but AI_COMPLETE and AI_FILTER returned NULL. So add a check for NULL outputs, as shown after this list.
• Fix the input to retry a failed row. return_error_details => TRUE shows the cause, here max tokens of 131072 exceeded (error details). A no-op UPDATE ... SET body = body does not retry the row. For an active row, an input fix does retry it.
• CREATE OR REPLACE reinitializes every dynamic table downstream. CREATE OR ALTER does not (best practices).

For the NULL check, a data metric function counts the NULL summaries each hour. The expectation reports a violation when the count is not 0. This needs Enterprise Edition.

ALTER DYNAMIC TABLE ticket_enriched
  SET DATA_METRIC_SCHEDULE = '60 MINUTE';

ALTER DYNAMIC TABLE ticket_enriched
  ADD DATA METRIC FUNCTION SNOWFLAKE.CORE.NULL_COUNT ON (summary)
  EXPECTATION no_null_summaries (VALUE = 0);

What you should see: in my test, one NULL summary made SYSTEM$EVALUATE_DATA_QUALITY_EXPECTATIONS return the value 1 and EXPECTATION_VIOLATED as TRUE.

Wrap-up: both answers are yes

“Can we improve a prompt without paying for every ticket again?” Yes. My prompt change re-read the 16 active rows and none of the 10 frozen ones, with no backfill script. Between prompt changes, the model reads only new and edited tickets. So a prompt change costs your active region, not your history.

“Can we improve a prompt without changing last quarter’s numbers?” Yes. 0 of the 10 frozen summaries changed. In the active region, 15 of the 16 summaries changed.

These numbers come from one demo run with 27 synthetic tickets. Yours will differ. The shape won’t.

Pick one scheduled enrichment job, ideally the one with a history table and a backfill script. Then do three things:

1. Move its SELECT into a dynamic Iceberg table with REFRESH_MODE = INCREMENTAL.
2. Freeze the rows that are complete.
3. Add the NULL check.

If you have no such job yet, the same layer fits anywhere a current-state Iceberg table sits next to free text:

• Network operations: classify technician notes by fault type (AI_CLASSIFY).
• Clinical data products: summarize site-monitoring notes in one line (AI_COMPLETE).
• Perception engineering: flag annotator comments that report a labeling problem (AI_FILTER).
• Enterprise reporting: classify service and invoice descriptions into your report categories (AI_CLASSIFY).

Next Monday, improve the prompt. On Tuesday, the refresh reads your active rows, not your history. On Wednesday, last quarter’s report still matches the dashboard. On Thursday, both answers are yes. On Friday, another team reads the same table from its own engine, with the same grants and no export.

A backfill’s bill grows with your history. This table’s bill grows with your changes.

The code in this post is sample code, provided as-is without official support from Snowflake.

Big thanks to PD Dutta and his Engineering team for their controbutions to this post.

The Missing AI Layer in Your Iceberg Lakehouse was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

When Databricks announced that it was launching a CDP product, it created a frenzy of activity on LinkedIn. It was an unusual move that created more questions than answers. Every expert chimed in with their two cents, and few added anything beyond hot takes. Now that the dust has settled a little bit, let’s give this a closer look.

To understand why the announcement landed the way it did, it’s worth remembering that the market was just barely coming to terms with the fact that traditional CDPs could barely keep their heads above water. The composable, zero-copy and data-warehouse-native approach was becoming the de-facto approach in a backlash against years of never-ending and expensive CDP projects that had mostly failed to deliver clear ROI. Even that composable approach had started to show its limitations and most activation products (Campaign Management) had muddied the water by professing that they had a built-in CDP. So it quite rightly got a lot of people wondering whether the Databricks announcement was yet again going to upset the applecart.

It’s not a bad question. Composability is pragmatic and can enable businesses to make the most of what they have invested in, as well as build a “best of breed” architecture. At the same time, native functionality can bring architectural advantages in terms of closeness, more effective data transfer protocols and overall cost. Put simply, composing a stack with too many tools means as many UIs, as many contracts, as many integration points, additional maintenance, different tech stack, etc. Lately it’s also become very clear that AI and decisioning had to be somewhat central. There is a point at which the downside of composability outweighs the benefits. As proven by the trail of failed projects, a CDP just by itself has little value except when part of a broader solution that supports use-cases with value and ROI. So yes, why not embed it into Databricks?

So businesses that have selected Databricks as their data-warehouse of choice can now activate a CDP capability. It’s an easy choice in principle, but so far the solution clearly does not have the breadth of capabilities of a traditional CDP (notably in terms of real-time capabilities), even if it could be “enough” for a number of use-cases. The unintended side effect is that it could discourage third party vendors to work with Databricks. Clearly for businesses like Hightouch, it looks like an 80% overlap of capabilities so that relationship is unlikely to thrive. After a few years of filling the reverse ETL gap, Hightouch is getting squeezed by its former partners, Databricks on one side and Braze/Iterable on the other.

As illustrated above, all the solutions adjacent to the CDP have for years been edging in its direction, encroaching on capabilities and increasing the overlap between solutions. So Databricks launching a CDP was not really surprising from that perspective. But the news that really threw me was the announcement that Bloomreach was a launch partner! I remember my remarkably eloquent reaction at the news: “Wait, what?” If you don’t see the massive irony in this announcement, I’ll remind you that the foundation of Bloomreach was the Exponea CDP, a positioning that is still being held through paid SEO and reflected on their website. I know vendors get creative with their positioning but this is reaching unprecedented levels of dissonance.

I wondered what the Bloomreach customers who have been sold a CDP are thinking of this… Using Databricks’ CDP to feed your campaign management’s CDP sounds ridiculous and maybe the term CDP is being misused on both sides. But the issue of overlap in orchestration and segmentation is real. It also exists with reverse ETLs, and once again it is staring straight at us as if asking the market “What you gonna do about it?”

The question to ask yourself is, “Who is actually the user?” or more pointedly “Who owns segmentation and audience building?”. This question highlights the architectural issue with most of the leading CRM solutions: the synchronisation between a fundamentally relational model and a proprietary NoSQL, schema-less customer + event store.

According to Databricks’ documentation on the underlying multi-agent approach, Genie translates the natural-language brief into a precise query, providing the SQL logic behind it for full transparency. So anyone without SQL knowledge can create a query that they’ll be incapable of checking for accuracy. I have no doubt that the “Find me people who haven’t bought in the last 2 months” will work, but what will happen when this is extended to real life scenarios? When the request spans over 4 or 5 tables, needs to take into account the total amount of spend on products in a specific category, exclude transactions that were returned (which can be in a separate table), etc.

Independent benchmarks tell a sobering story. BIRD (a benchmark built from real, messy databases) puts leading models at 73% accuracy. When moving to enterprise-scale complexity (Spider 2.0’s multi-step, agentic, advanced schemas), accuracy falls to 21.3%. Take a few seconds to let those numbers sink in… What head of CRM is going to launch a campaign with an audience that has this level of accuracy? We’re not talking about affinity scoring, we’re talking about messages or journeys that will only make sense if the customer is in the right audience. Expected accuracy is 100%! If it isn’t, someone’s made one of those mistakes that used to get you fired.

Full disclosure: this is where imagino plays. I truly believe that a Query Builder, like the one provided by imagino, is currently the best way to bridge the gap between CRM users and the SQL world. It allows the user to understand the query logic and to tweak the parameters, even when they start from natural language. The truth is that the user of Databricks’ CDP won’t be in marketing, they’ll be in IT and that’s ok as long as expectations are set accordingly.

To conclude this article, I’ll try to briefly tackle one of the underlying questions, specifically from a data-warehouse perspective: does it provide Databricks an edge or is it a distraction from its core focus. Snowflake has always privileged its partner network and constantly invests in those relationships. Databricks has probably made a few enemies but most importantly, that’s a totally different product line that they are not experienced in selling, and a product that they’ll need to defend, maintain and improve. I’m not saying it won’t be fit for purpose in some IT teams but I do doubt that they will convince the marketing crowds.

From a Snowflake perspective, there are two important points to remember:

• They have created a formidable ecosystem of solutions, CDPs just being one of them.
• Snowpark enables some of those solutions to be run fully as Native Apps.

Bearing in mind those two points, I think Snowflake will remain undeterred by the move. There is little in Databricks’ CDP that cannot be achieved by a solution like imagino, purchased via the Snowflake Marketplace, and deployed as a native app. Very little.

Mathieu Lavedrine has spent his career at the intersection of martech and data. 8 years at Neolane/Adobe, Tinyclues (a pre-LLM AI company), and now running partnerships and competitive strategy at imagino, a Snowflake-native CDP/CEP.

Originally published at https://www.linkedin.com.

A Martech Perspective on Databricks’ New CDP Capability was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

During the proof of concept for this post, one of our scheduled agent runs reported success. The result was wrong. The task did not fail, so the run state had nothing to say about it. The agent had simply done the job wrong, on a schedule, with nobody watching.

That was the moment I stopped thinking of agents as a chat window. An agent that runs at 3am, reads tables, and writes something somewhere is a pipeline step. And every other pipeline step on my platform is versioned, reviewed, and deployed through a plan. This post is about giving agents the same treatment.

In Part 1 of this series I listed “an automation running a nightly health check” as one of the reasons agent SQL is spiky. In this part that automation is the subject.

When the agent stops being a chat

The first three parts of this series treated agents as consumers. They burn compute (Part 1), they need an organization that governs them (Part 2), and they need budgets and quotas (Part 3). This part treats them as producers: something that runs on its own and changes things.

The tooling is now mature enough for that question to be real. Cortex Code (CoCo) CLI has been generally available since February 2026, CoCo in Snowsight since March 2026, and CoCo Desktop since July 2026. In the last few months, Snowflake added the two pieces that move CoCo from “a session I am watching” to “a job that runs without me”: automations and Cloud Agents.

Like Part 1, this was a proof of concept, not production. We tried CoCo automations, Cloud Agents, the CLI --cloud mode, and DCM Projects (generally available). DCM, short for Database Change Management, is Snowflake's way to describe objects as code in definition files and deploy them through a plan you review first. We used them on three kinds of work:

• Health and data quality checks on a schedule
• Drafting dbt models or DCM definitions for a human to review
• Investigating failed pipeline runs

I am keeping the scale qualitative. What matters here is not how many runs we did, but what the defaults did to us.

CoCo automations: an agent on a schedule

CoCo automations have been in preview since August 2026. An automation turns a saved prompt into a recurring, unattended CoCo run that executes in a Snowflake-managed sandbox, even when your terminal or browser is closed. Each run creates a Cortex thread you can inspect and continue later. An automation created in the CLI shows up in Snowsight and the other way around, because both create the same Snowflake object.

That object is the part a platform engineer should look at first. Each automation is stored as an AGENT TASK, a type of Snowflake task, in the PUBLIC schema of the user's personal database (USER$.PUBLIC), with an internal name that starts with COCO_ROUTINE_. It is owned by the user, not by a role. It needs no warehouse. By default the user's USER$.PUBLIC.DEFAULT$ stage, a file area in that personal database, is mounted at /workspace, the run's working directory, so files persist between runs.

Creating one from the CLI looks like this. This is the example from the documentation:

cortex automation create \
  --name pipeline_health \
  --prompt "Check pipeline failures from the last hour and summarize the likely causes." \
  --schedule "every 60 minutes"

Two options matter for reproducibility. --prompt-file reads the prompt from a file instead of the command line, and --dry-run prints the generated task information and SQL without creating anything. Schedules are fixed and time based, such as daily at 9am or every Tuesday at 1:15pm with an IANA time zone. The minimum supported frequency is hourly, and there are no event triggers yet.

CoCo Desktop also has automations, but they are a different thing: saved prompts stored on one machine that run only while the app is open and the computer is awake. For pipeline work, I only consider the Snowflake-hosted kind.

Cloud Agents: the same agent, off your laptop

Cloud Agents are generally available in all commercial regions on AWS, Azure, and Google Cloud. Every CoCo session in Snowsight now runs in its own Snowflake-managed container, which means CoCo in the browser can run shell commands and Python scripts, and search the web once an ACCOUNTADMIN enables it.

The boundaries are what make this usable for pipeline work. The filesystem is scoped to the session. Outbound network access is restricted, not disabled: outbound HTTPS traffic (egress) only reaches an allowlist of package managers and build tooling, and only for reads, so installing a package works and publishing does not. External access integrations are not supported. Cloud Agents does not change any RBAC (role-based access control) settings or add SQL access, and for now it comes at no additional cost, with pricing possible in a future release.

The CLI has the same idea behind --cloud, which is generally available and which the docs describe as the CLI equivalent of Cloud Agents in Snowsight. Tools run in a Snowflake-managed container, your local files are never visible to the agent, and only the mounted /workspace persists. Everything written anywhere else is discarded when the session ends.

In the POC we used cloud sessions for two jobs: investigating failed runs, and drafting dbt or DCM code that a person would then review.

What surprised me

Three things caught us in the POC, and all three are documented. We just had not read that part of the page carefully enough.

Runs used the creator’s default role. Each automation run starts a task session whose primary role is the user’s default role, plus their default secondary roles. It is not the role that was active when the automation was created, and that role is not recorded on the task. So an engineer whose default role is powerful can create an automation in a careful session with a narrow role, and the automation still runs with the powerful one.

Everyone could schedule agents. Access to automations is controlled by the EXECUTE AGENT TASK account privilege, and it is granted to PUBLIC by default. In a fresh account, every user can schedule an unattended agent.

Nobody approves anything, and success is not correctness. Automation runs are unattended, so interactive tool permission prompts are disabled. That is the run I opened this post with: the task succeeded and the outcome was wrong. The documentation is direct about this. Task history is the authoritative source for run state and errors, but a successful task can still produce an incomplete business outcome. The transcript is where you find out.

The defaults I changed

None of these changes are clever. They are the same controls I would put on any service that can run SQL unattended.

First, the privilege. I revoked it from PUBLIC and granted it to a role for the people who are allowed to own automations. This is the documented pattern:

REVOKE EXECUTE AGENT TASK ON ACCOUNT FROM ROLE PUBLIC;
GRANT EXECUTE AGENT TASK ON ACCOUNT TO ROLE automation_user;

Second, the role a run actually uses. Because a run takes the owner’s default role, the users who own automations got a narrow default role. That role decides what a 3am run can touch, so it is the one to design, not the role in someone’s open session.

Third, interactive chats. User-managed Restricted Session Scope became generally available in September 2026 in CoCo CLI, CoCo Desktop, and CoCo in Snowsight. It restricts a single CoCo chat to read-only access or to selected roles, and it can only narrow privileges, never add them. We used read-only scope for chats. One caveat: the CLI --cloud mode does not support restricted session scope or --sql-read-only, and the CLI exits with an error if you combine them.

Fourth, the network. An ACCOUNTADMIN can block all non-Snowflake egress from Cloud Agents with the account parameter COCO_CLOUD_AGENTS_NON_SNOWFLAKE_EGRESS_DISABLED, which defaults to FALSE. We set it to TRUE.

Cost is Part 3’s story, so one line here: during the preview, every automation run incurs standard task billing plus CoCo token consumption. And as in Part 1, automation runs set a QUERY_TAG, so their queries can be traced by tag like every other pipeline.

Keeping agent work reproducible

Locking down defaults makes agents safer. It does not make their work reproducible. For that I use one rule: the agent proposes, the pipeline deploys.

An agent may look, check, and draft on its own. Anything that changes an object in Snowflake goes the same way a human change goes. The agent works on a branch, the change goes through a GitLab merge request, CI runs a DCM plan, and the deploy happens only after someone approves it.

DCM Projects are what make the last two steps work. They became generally available in August 2026. You describe the target state of databases, schemas, tables, tasks, and other objects in definition files with DEFINE statements, use Jinja for anything that differs between environments, and Snowflake works out the changes. Definition files can live in a Snowflake Workspace, a remote Git repository, or a local directory, and the docs recommend Git for versioning.

In CI, the plan step is the review artifact. In the Snowflake CLI form from the docs:

snow dcm plan --target PROD_US --save-output
# review the plan in the merge request, then after approval:
snow dcm deploy --target PROD_US

snow dcm plan shows what would be created, altered, or dropped without applying anything, and --save-output writes the result to the local out/ directory so CI can attach it. The documentation is blunt about the order: always run and review your PLAN output before running DEPLOY. It also notes that a successful plan does not guarantee a successful deploy, because DEPLOY computes a fresh changeset against the current account state.

Three more details make this reproducible rather than merely reviewed. Each DCM project object stores the immutable artifacts and definition files of every deployment, and SHOW DEPLOYMENTS IN DCM PROJECT lists them. The docs recommend one DCM project for each target environment, with the same definitions and different template values. And they recommend a separate owner role for production, held only by service users. That last one is the agent rule in different words: the agent never holds the production role.

Snowflake ships reusable GitHub Actions for DCM plan and deploy, but they are in preview and only available for GitHub. We run on GitLab, so those actions do not apply to us. The DCM TEST and PREVIEW commands are also still in preview.

The agent’s own instructions need versioning too. A prompt can live in a file and go in with --prompt-file, and --dry-run shows the task SQL before anything is created. Project skills for the CoCo CLI live in .cortex/skills/ inside the repository, so they travel with the code. Skill and plugin sharing across an account, through Cortex Extension objects, has been in preview since July 2026. What the automation page does not describe is any way to export or version an automation itself, so my recommendation is to treat a prompt file in Git as the source of truth, not the task.

If your repository lives in GitLab, a Git repository clone in Snowflake can sync it with all its branches, tags, and commits, and Workspaces can commit and push back to it.

Seeing what the agent did

The last piece is evidence. For every run I want two answers: did it run, and did it do the right thing. They come from different places.

Task history answers the first. The thread transcript answers the second: every message, tool call, result, and the final response. From the CLI, cortex automation doctor <name> shows recent runs and thread IDs, and cortex conversations transcript <thread_id> prints a run. Two details shaped how we check. A failing pre-run or post-run hook does not fail the run, so a green task can hide a broken hook. And threads and run history are kept for two months, so anything you need longer has to be kept somewhere else.

Options I considered

These are the options the documentation offers for getting an agent’s change into Snowflake, and how I weigh them against what the POC showed.

Let the agent deploy directly. It is the fastest path, and an automation can already run SQL. It lost because a run takes its owner’s default role, nobody approves its tool calls, and a successful task can still be the wrong outcome. A direct deploy leaves no plan for anyone to review.

Let the agent write directly in DEV only. This keeps the agent fast where mistakes are cheap. It lost because the docs recommend one DCM project per environment with the same definitions. A change the agent writes straight into DEV is not in those definitions, so DEV stops matching what reaches PROD.

Use Snowflake’s DCM GitHub Actions. They give plan and deploy steps out of the box. They are in preview and only available for GitHub, and we run on GitLab.

The agent proposes, the pipeline deploys. Branch, GitLab merge request, a DCM plan in CI, deploy after approval. It is slower than a direct write, and that is the point: every change has a plan someone approved.

Impact

This is what the setup makes possible for a team that adopts it. Reviewers see an agent change like any other change: a merge request with a plan attached, not a transcript to reverse engineer. Whoever is on call has one place, the transcript, to check that a green run did the right thing. For platform and security, the defaults we changed in the POC (revoking the PUBLIC grant, narrowing the default role, blocking non-Snowflake egress) close off the case where any user can schedule an unattended agent with a powerful default role and open egress.

It also changes the next decision a team faces: whether agents can move past a POC. With the same review path as human changes, that becomes a question about what the agent may propose, not about trusting it.

For a team starting out, the order matters: read the defaults section before creating the first automation, and design the default role first.

Durable definitions, variable agents

This series started with compute and ends with agents, and the principle stayed the same. In Part 1 I wrote about durable models on variable compute: hold the model of the workload steady and let compute flex underneath it.

Agents are the most variable thing I have put on the platform. The same prompt can produce a different plan tomorrow. So I hold everything around them steady: the role they run as, the network they can reach, the prompt in Git, and above all the path to production. The agent can explore, check, and draft as much as it likes. What reaches production is a reviewed definition and a plan someone approved.

A concrete place to start, before anyone schedules a first automation: check who holds EXECUTE AGENT TASK in your account (by default, PUBLIC does), and check the default role of every user who will own an automation, because that is the role a 3am run uses.

Running Snowflake in the Agent Era

Part 1: Stop Picking T-Shirt Sizes: What Running Adaptive Compute Taught Me

Part 2: From ORGADMIN to Control Plane: Organization Hub, Organization Users, and Premium Views in Practice

Part 3: Budgets, Quotas, and Monitors: The Cost Controls I Map Before Agents Go Wider

Part 4: The Agent Is Now a Pipeline Step (you are here)

The Agent Is Now a Pipeline Step: Keeping CoCo Automations and Cloud Agents Reproducible was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Part 1 ended with a query I was proud of. It grouped QUERY_METERING_HISTORY by QUERY_TAG and told me, for every Adaptive Warehouse, who or what had spent the credits. Every pipeline and every agent session on the platform sets a query tag, so the answer was precise.

Then a colleague asked the obvious follow-up: “Great. What stops it?”

That query is a report. It tells me after the fact. An agent does not get tired at six in the evening, and a Cortex Agent’s orchestration tokens never show up as a warehouse query at all. So in the same AWS proof of concept I used for Part 1 (a POC, not production), I set out to map the cost controls Snowflake now offers against the questions I actually need answered: which team spent it, which person spent it, which agent spent it, and what changed that nobody predicted.

I tried four of them hands-on: budgets scoped by the cost_center tag, per-user quotas, cost anomaly alerts, and a budget on a Cortex Agent. The rest of this post comes from the documentation, and I say so where it matters.

Two kinds of spenders, two kinds of credits

In the POC, people and agent sessions spent on the same account. A person runs a dbt model or an ad hoc query. An agent does that too, but it also burns tokens that never touch a warehouse.

Cortex Agents (generally available since November 2025) show why. Orchestration is charged by the tokens it uses. Custom tools run on a warehouse and incur warehouse cost. One question from a user can produce both kinds of spend.

That split runs through every control in this post. Snowflake measures warehouse compute in credits and AI services in AI credits, and the documentation is explicit that the two are different units and cannot be added together. A per-user quota watches either warehouse compute or AI domains, never both. An anomaly monitor tracks one credit family. If you want one number per team, you will build it from two.

Four questions, four controls

This is the map I ended up with.

• Which team? Custom budgets scoped by a tag such as cost_center.
• Which person? Per-user quotas.
• Which agent? A resource budget on the Cortex Agent object.
• What changed that nobody predicted? Cost anomalies, and anomaly monitors for a narrower scope.

They differ in the way that matters most: what happens when a limit is crossed. Snowflake’s AI cost governance page describes per-user quotas as the only AI cost control with built-in enforcement. Budgets notify you and run the stored procedures you attach to them. Anomaly detection tells you once a day. Only the quota blocks.

Budgets by cost center, not by object

In Part 1 I tagged every Adaptive Warehouse at creation with governance.tags.cost_center (the tag name is illustrative). This is where that tag pays off.

A custom budget can track objects by a tag and value pair instead of by a list of objects. Every object carrying the pair counts, including objects that inherit the tag from a parent, such as tables in a tagged database. An object can count toward several budgets, so a central platform budget and a team budget can watch the same warehouse. New warehouses with the right tag join the budget without anyone editing it, which is the whole point: the budget follows the cost model, not the inventory.

GRANT APPLYBUDGET ON TAG governance.tags.cost_center TO ROLE budget_admin;
USE SCHEMA budgets_db.budgets_schema;
CREATE SNOWFLAKE.CORE.BUDGET data_platform_budget();
CALL data_platform_budget!SET_SPENDING_LIMIT(1000);
CALL budgets_db.budgets_schema.data_platform_budget!SET_RESOURCE_TAGS(
  [
    [(SELECT SYSTEM$REFERENCE('TAG', 'governance.tags.cost_center', 'SESSION', 'APPLYBUDGET')), 'data_platform']
  ],
  'UNION');

The limit here is illustrative. SET_RESOURCE_TAGS takes an array of tag references and values, plus 'UNION' (any pair matches) or 'INTERSECTION' (all pairs must match). It replaces whatever tags the budget had before, and a budget takes up to 20 resource tags by default.

Two timing rules belong in your runbook. After you change a tag on an object, it can take up to six hours to show up in tag-based budgets. And tag changes made in the first two days of a month are applied to the prior month’s usage, which will confuse anyone reconciling month-end numbers.

Budgets can also act. A custom action runs a stored procedure when spend reaches a threshold, based on projected or actual consumption, and a cycle-start action can undo it when the monthly cycle restarts. In the POC, that is where I met the first surprise: a budget action fired later than I expected. The docs explain why. Thresholds are evaluated periodically, and Snowflake’s AI cost governance page says budget thresholds take up to eight hours to act, or up to two hours with the latency optimized option. A budget is a lagging control. Plan for that.

Shared resources are different. When several teams use the same AI features, a budget can count only the credits used by tagged users. Budgets for AI features became generally available in April 2026. The same user-tag model extends to shared warehouses, where the budget tracks each user’s share of interactively issued query cost and leaves out idle time, very short queries, cloud services overhead, and automated work such as tasks.

Per-query showback with QUERY_TAG

Budgets tell me a cost center crossed a line. They do not tell me which pipeline or which agent session pushed it there. That is still the job of the Part 1 query.

For Adaptive Warehouses, QUERY_METERING_HISTORY returns per-query credits with USER_NAME, ROLE_NAME, WAREHOUSE_NAME, and QUERY_TAG. QUERY_ATTRIBUTION_HISTORY has no records for Adaptive Warehouses. Our agent sessions set QUERY_TAG exactly as Part 1 describes, so I did not need anything new here.

Snowflake’s cost attribution guidance draws the same line I do. Use object tags for resources and users owned by one group. Use query tags when one application runs queries on behalf of several departments, which is exactly what an agent serving many teams does. The object tag sets the boundary. The query tag explains who crossed it.

Per-user quotas: the only control that blocks

Per-user quotas became generally available in August 2026, and weekly limits followed in September. A quota is an object (SNOWFLAKE.CORE.QUOTA) with a scope of users, a per-user limit per cycle, and an enforcement setting.

USE SCHEMA cost_mgmt_db.quota_schema;
CREATE SNOWFLAKE.CORE.QUOTA ai_quota();
CALL ai_quota!ADD_SHARED_RESOURCE('CORTEX AGENT');
CALL ai_quota!ADD_SHARED_RESOURCE('CORTEX CODE');
CALL ai_quota!SET_PER_USER_LIMIT(120, 'DAILY');
CALL ai_quota!SET_BLOCK_ENFORCEMENT_ENABLED(TRUE, TRUE);

Again the limit is illustrative. Cycles are daily, weekly, or monthly, all aligned to UTC. By default a quota covers every user in the account, and you can narrow it with user tags or name users to include or exclude.

Three details changed how I think about quotas.

Limits are per user, never pooled. Every user in a quota gets the same limit, and there is no collective cap. Snowflake’s own example: a 100-credit quota across 10 users permits up to 1,000 credits. A quota protects you from one runaway user or agent session. It is not a team budget.

Blocking covers AI domains only. Block enforcement applies to AI functions, Cortex Agents, Snowflake CoWork, CoCo (generally available), and AI Gateway (in preview as of September 2026). You can add warehouses to a quota and track them, but block enforcement does not apply to warehouses.

Blocks are fast, not instant. The docs say evaluation is not real time but occurs within minutes, and that spend can overshoot. Their example is a 100-credit limit where a user ends at 130 before the block lands. That matches what I saw in the POC: blocks kicked in later than I expected and spend went past the limit. Add the cycle boundary, where a user can spend a full limit just before the reset and another just after, and you see why the limit should sit below what you can actually tolerate.

When someone legitimately needs more, the documented pattern is a second quota with a higher limit, scoped with an INTERSECTION of user tags, rather than raising the limit for everyone. CoCo also has its own per-user daily credit limit parameters if you want a separate control for that surface.

A budget on the agent itself

A quota limits people. Sometimes I want to limit the agent: a monthly cap on one Cortex Agent, whoever calls it. That is a resource budget for Cortex Agents.

The setup reuses the tag pattern. Tag the agent with ALTER AGENT … SET TAG, create a budget, set a monthly limit, and associate the tag with SET_RESOURCE_TAGS. To make the budget act, you register a stored procedure with ADD_CUSTOM_ACTION that revokes a dedicated agent role at 100 percent, and a cycle-start action that grants it back next month. You can set thresholds up to 500 percent, which lets you reinstate a few users and still keep a hard stop higher up.

ALTER AGENT IF EXISTS my_agent
  SET TAG governance.tags.cost_center = 'data_platform';

In the POC I put a budget on a Cortex Agent, and it behaved like every other budget: the action came later than I wanted. The docs say enforcement can take up to eight hours after the budget is exceeded, or about two hours with the low latency option, and they recommend an early alert at 80 percent. I now treat that 80 percent notification as the real control and the revoke as the backstop.

Two gaps to plan around. Revoking only works if the agent has a dedicated role and users have no other path to it. And requests that start in Snowflake CoWork and call the agent are billed to CoWork, so a budget scoped only to the agent’s tag does not see them. Cover CoWork in the same budget or give it its own.

Catching what nobody predicted

Budgets and quotas only catch what I thought to limit. Cost anomalies catch the rest.

Cost anomaly detection (generally available since December 2025) compares each day’s consumption with the expected range based on prior consumption. It is always on at account and organization level, with no setup. It needs at least 30 days of history, and it stays quiet when the last seven days were under 10 credits. Since July 2026 the notifications can also go to Slack, SMS, or a webhook through a notification integration.

In the POC an account-level anomaly alert caught something real, something none of our budgets or quotas had been set up to watch. That is exactly the job I want from it.

Anomaly monitors (in preview as of August 2026) narrow the scope. A monitor is defined by object tags and service types, so you can watch one cost center instead of the whole account. Each monitor tracks one credit family, credits or AI credits, so one cost center that spends both needs two monitors. An account can have 20 monitors, monitors work in a single account, they recompute daily, and they cannot split a shared resource’s spend by user. I have only read about monitors so far.

The order I would roll it out

This is a recommendation, not the order we followed in the POC.

1. Tag first. Warehouses, agents, and users. Every other control reads tags.
2. Budgets per cost center. Tag-scoped, with an early notification threshold.
3. QUERY_TAG showback. So a crossed budget leads to a pipeline or session, not a guess.
4. Per-user quotas with block on AI domains. Limits below what you can tolerate, because of overshoot.
5. Resource budgets on the riskiest agents. Dedicated roles, an 80 percent alert, CoWork covered.
6. Anomaly alerts routed to the team channel. Then monitors per cost center once you trust them.

What these controls do not do

None of these is a hard real-time limit. Budgets act on a delay of hours. Quotas act within minutes and overshoot. Anomalies arrive daily and need 30 days of history.

Quotas cannot block warehouse spend. For warehouses, the resource monitors from Part 1 are still the control that stops intake, and on an Adaptive Warehouse they disable it rather than suspend it.

There is one open question I could not settle from the docs, and I would like you to test it. The shared warehouse budget page says it attributes user cost using QUERY_ATTRIBUTION_HISTORY boundaries, and that view has no records for Adaptive Warehouses. Neither that page nor the per-user quota page says whether user-tag budgets or the warehouse domain of a quota cover Adaptive Warehouses. If you run them on Adaptive Warehouses, compare their numbers with QUERY_METERING_HISTORY before you trust them.

Durable cost models on variable spenders

Part 1 closed on durable models on variable compute. Part 3 is the same idea one level up. The tag is the durable model. People and agents are the variable spenders. Budgets follow the tag, quotas follow the person, resource budgets follow the agent, and anomaly detection watches for what none of them expected.

The lesson from the POC was not a number. It was learning which control acts, how fast, and which one only reports.

Budgets, Quotas, and Monitors: The Cost Controls I Map Before Agents Go Wider was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Snowpark Python’s DataFrame.to_polars() is now generally available, bringing Snowflake query results into a Polars DataFrame with a single call. Its default Parquet transport reads files in parallel to accelerate large data transfers, connecting data preparation in Snowflake with analytics and data processing in Polars.

Why it matters

Snowpark already supports pandas through to_pandas(). With to_polars(), we are extending that interoperability to developers who use Polars in their Python workflows.

You can use Snowpark to filter, join, and aggregate large datasets in Snowflake, then return the results as a Polars DataFrame for further processing. This lets you combine Snowflake’s data processing with your existing Polars application, using the same Snowpark session.

How to use it

With a configured Snowpark session and Polars installed, call to_polars() on a Snowpark DataFrame. Here is a small example:

import polars as pl
from snowflake.snowpark.functions import col

snowpark_df = session.create_dataframe(
    [["North", 120], ["South", 80], ["North", 200]],
    schema=["REGION", "REVENUE"],
).filter(col("REVENUE") >= 100)

polars_df = snowpark_df.to_polars()

# Continue processing with Polars.
summary = polars_df.group_by("REGION").agg(
    pl.col("REVENUE").sum()
)
print(summary)

In this example, calling to_polars() executes the Snowpark query, filtering out rows with revenue above 100 in Snowflake and returning the remaining rows as an eager Polars DataFrame. Polars then groups those rows by region and sums their revenue, producing one result: North with a total of 320. South is excluded by the filter.

Note

DataFrame.to_polars() uses Parquet transport by default. Snowpark unloads query results to the session stage as Parquet files and reads them in parallel into Polars. This can speed up large, transfer-heavy workloads, especially in Snowpark Python stored procedures.

Use to_polars(transport="arrow") when you need to preserve timezone-aware timestamps or full timestamp and floating-point precision. Parquet does not support TIMESTAMP_LTZ or TIMESTAMP_TZ columns, rounds TIMESTAMP_NTZ values to milliseconds, and reduces FLOAT precision to float32.

Both transports return a fully materialized DataFrame, so the result must fit in your Python environment’s memory. Filter rows and select the columns you need in Snowpark before converting; subsequent Polars operations run in your Python environment.

Get started

Use Snowpark Python 1.55.3 and Polars 1.0 or later. Start with a Snowpark DataFrame from your own workflow and call to_polars() to continue working with the results in Polars.

Explore the API reference for parameters and transport guidance.

Snowpark Python Supports Native to_polars() was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

Salesforce’s journey since ExactTarget

Salesforce bought ExactTarget 13 years ago for $2.5B and it has turned out to be a great investment. The product now called Engagement has been drifting towards its end of life for a while. The numbers don’t lie, Marketing & Commerce has been the weakest-performing division (1.5% growth in Q4 FY2026, and is now conveniently folded into the Agentforce number).

Salesforce has tried to address some of the technical limitations with Data Cloud (later renamed Data 360) but it is still a separate foundation from which data needs to be synchronised and duplicated. On top of this architectural challenge, Data 360 comes with a notoriously eye-watering price tag and obscure pricing. A customer reported to me 7x the predicted spend after two years of operations. Perhaps they did more than planned, perhaps it wasn’t implemented properly, etc. Whatever the reason, they didn’t see it coming and it’s the type of seven-figure problem no CMO wants on their CV.

Finally, the Marketing Cloud Next product that Salesforce has been building for a while admittedly doesn’t have functional parity with Engagement. Buying a whole product instead, especially on the scale of Braze, looks like admitting defeat.

It is reminiscent of when Adobe stopped investing in Adobe Campaign Standard (its attempt to rebuild Neolane as proper SaaS) and restarted from scratch with AJO (Adobe Journey Optimiser) which is still far from offering functional parity with Adobe Campaign Classic (previously known as Neolane) even if it does other things better. AJO also comes with the heaviness of AEP (Experience platform) and that event-driven architecture which is expensive to run and ill-aligned with high data volumes and complexity. An interesting parallel that suggests the challenge is not trivial.

Braze’s architectural sword of Damocles

On the Braze side lies the uncomfortable fact that even with the best will in the world, a SaaS solution cannot reinvent its architecture. Braze is a great front-end that has experienced substantial growth over the years, benefiting from the innovation void created by Salesforce and Adobe resting on their laurels. Yet, its power is also its greatest weakness. The event-driven, profile-centric architecture has inherent limitations by design. It made the platform simpler to configure and use, which, combined with a simple UI, attracted numerous business teams. Of course there is a price to pay for simplicity; anything slightly advanced from a data perspective had to be done ahead of being synchronised into Braze.

That data gap has become more apparent in recent years because both volume and complexity of data have increased. Adding a tool like Hightouch looked like the perfect solution but quickly the segmentation and orchestration friction could not be ignored and the unofficial alliance ended. With tools like Iterable and Klaviyo, Braze now sits in a class of solutions that come with inherent architectural limitations. It’s a challenge when the market is moving towards data-warehouse-native architectures in an effort to reduce data pipelines and their associated costs and complexity.

For Braze, it’s the epilogue to an incredible success story and maybe the realisation that they were running out of ideas on where to go next. Despite strong revenues, they were trading at $30, five years after an IPO at $65. If I were an investor I’d be quite pleased with the $47.50 buyout price, cut my losses and walk away.

What should CMOs do if they were about to buy either solution?

I was hesitating between “running for the hills” and “pray and hope the integration will be smooth”, but as appropriate as it might be, I realise it’s not helpful. I have been both on the acquired and the acquiring side in the past, so I’m fairly confident when I say that both companies will try to reassure and claim that nothing changes for the time being. It’s usually true. An integration plan of this magnitude is extremely complex and will require a lot of time and energy. While it’s somewhat reassuring, it’s also frightening to see how an acquisition of this type is like slamming on the brakes of innovation and goodwill. I am saying this from first-hand experience, referring to a product that I am still deeply fond of, and was sad to see die a slow death.

Don’t get me wrong, those things don’t happen from bad intentions. The “simple API integration” is never that simple, the multiple codebases are not just different languages but a different ethos, the dev teams clash, the good people give up, the rest fight for control or leave with a golden parachute. Just look at what happened to Unica, Silverpop or EmailVision. The martech graveyard is full of once-strong solutions that have been acquired, renamed, sold off for scraps, or retired into a PE fund for a few more years of revenue extraction. For now, expect two businesses that were fiercely competing with one another last week to have everything now surprisingly making sense in a new markitecture where Data 360 is as useful a foundation to Braze as it’s supposed to be for ExactTarget/Engagement, Pardot or Next…

In practice, if you’re signing a two-year term, be ready for a re-evaluation at the end and pay particular attention to change-of-control and termination clauses, price protection on renewal, written roadmap commitments, and as always push for a proof of concept. If you’re planning for the longer term, it might be worth looking again at what type of business you want to partner with, and what architecture they are built upon. I’m not biased enough to believe imagino is right for everyone, but I am deeply convinced that the architecture it provides needs to be positioned against what Adobe/Salesforce/Braze have to offer. If your review process has not looked into this, it should, urgently.

What it means for the market

Let’s remember this might just be a market share move; it financially makes sense for Salesforce to buy the company where a lot of their clients were going. That being said, I don’t believe it can resolve the underlying issue with the architecture of either stack. Mary Shelley would be impressed by a Frankenstack made of Pardot, Krux, Datorama, Informatica, ExactTarget, Evergage and Braze just to name a few. And I’ll refrain from scratching at the AI paint job that brings all of this to life. Ultimately, there is also a certain lack of love for products that are sold to the C-suite and which business teams are left struggling with. Marketing teams will take stock, and many agencies will be disappointed to see a valid option being gobbled up by a company like Salesforce.

The good that will come from this is the forced re-evaluation of the market. If data ownership is important and you’ve built a foundation on BigQuery, Microsoft Fabric or Snowflake, avoiding Adobe or Salesforce’s data clouds now leaves few enterprise options on the table, imagino being one.

Mathieu Lavedrine has spent his career at the intersection of martech and data. 8 years at Neolane/Adobe, Tinyclues (a pre-LLM AI company), and now running partnerships and competitive strategy at imagino, a data-warehouse-native Customer Experience Platform.

What does the Braze acquisition by Salesforce mean for the market? was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR: Optima Planning already learns from repeated queries. It can now also learn across queries — when a brand-new query shares a plan fragment or join with something that has already run, it reuses that feedback on the very first execution. In production, this turned a 19-join analytics query from almost two hours into 30 seconds. No manual configuration required.

Optima Planning: One more question about a query

In a production workload by a large U.S.-based healthcare technology company, cross-query feedback reduced a 19-join analytics query from almost two hours to 30 seconds. Here’s how that’s possible.

Optima Planning has so far answered one question about a whole query: has this query, or a variation of it, run before? (Covered in our previous “Boosting Recurring Query Performance with Snowflake Optima Planning” blog.) Optima Planning can now also answer a complementary question about a piece of the plan inside a query — a plan fragment, or an individual operator.

Per-query feedback answers an important question:

Has this query, or a variation of it, run before?

Matching on a plan fragment or an individual operator answers a complementary question, about a piece of the plan rather than the whole query:

Has this plan fragment, or this specific operator such as a join, appeared inside any other query that has already run?

Together, they widen the population of queries Optima Planning can learn from, covering both queries that are the same and queries that repeat in shape but are not the same.

How this works

Today, Optima Planning can also match inside the query — on a plan fragment, such as the join tree inside a shared view, or on an individual operator, such as a specific join — at a finer grain than the whole query. At a high level:

• After a query compiles, Optima Planning computes a hash for each plan fragment and each individual operator in the plan, such as a specific join, over the optimizer’s output for that piece of the plan, not the whole query’s SQL text.
• If a different query later contains a fragment or operator with that same hash, even if the rest of that query is entirely new, such as an added join to a table that has not been joined before, Optima Planning recognizes the match on that shared piece.
• Feedback from the first query’s execution feeds into the second query’s plan for that shared fragment or operator, so the optimizer chooses its plan with evidence for that piece instead of starting cold, even though the second query has not run before as a whole.

Matching on a plan fragment or an individual operator finds a shared piece of the plan. That match is a candidate, not a commitment. Optima Planning also keeps track, intelligently, of whether reusing that feedback from execution in the new query would over-generalize: applying evidence from one context in another where it no longer holds. When the transfer looks too aggressive, that feedback is not applied.

The important part is that the match happens on a fragment or an operator in the plan Snowflake’s optimizer produces, not on the whole query’s SQL text.

This does not replace per-query feedback, which already handles variations of the same query. It is intended to complement it, filling in for the case that per-query feedback cannot reach: a query that is not the same, because it is built on a fragment or operator that has already run inside some other query. The handoff between the two mechanisms is clean: matching inside the query helps on that new query’s first execution. The moment that same query runs a second time, it has its own identity like any other repeated query, and per-query feedback takes over from there.

Snowflake’s optimizer is cost-based: it estimates each operator’s row count and picks the plan expected to be cheapest. When an estimate is wrong, the optimizer can pick the wrong join order or build side, sometimes cascading into billions of unneeded intermediate rows (covered in more depth in “Boosting Recurring Query Performance” and “Grounding Query Plans in Real Data”).

Where learning across queries helps

Extending an existing view with a new join

-- v_customer_orders already joins customers, orders, and order_items, and already has feedback
SELECT * FROM v_customer_orders v JOIN returns r ON v.order_id = r.order_id WHERE ...  -- new join, never run before

An analyst builds on a view that’s already common in the workload, adding one join this workload has not run in that combination before. The combined query is brand new and gets a brand-new identity; the join fragment inside v_customer_orders is not new at all.

A common join pattern reused in a new report

-- the customers-orders join already runs inside a dozen other reports
SELECT c.region, COUNT(*) FROM customers c JOIN orders o ON o.customer_id = c.customer_id
WHERE o.status = 'RETURNED' GROUP BY c.region  -- this exact report is new

The specific report, this exact SELECT list and grouping, is new and has never run before. The join it depends on, customers to orders, is not new: it already has feedback from every other report that uses it.

Logically identical queries, written differently

FROM orders o JOIN customers c ON c.customer_id = o.customer_id   -- one BI tool's join order
FROM customers c JOIN orders o ON o.customer_id = c.customer_id   -- a different tool, same result

Two BI tools can generate the same join with the tables listed in opposite order. For now, those are treated as different queries. Matching on a plan fragment or an individual operator can still share feedback on that join, because it looks at the operator in the plan rather than the FROM-list order in the text.

In each case, matching inside the query can reuse feedback already learned on that shared fragment or operator.

A synthetic example

Let’s reuse the restaurant example from our earlier blog: the restaurants, menu_items, sourcing_records, and suppliers tables, joined on restaurant_id, ingredient_category, and supplier_id. As we saw earlier, top-rated restaurants tend to use a small set of premium ingredients, and those ingredients tend to come from artisan suppliers rather than industrial ones. That pattern is spread across three tables, so per-table statistics have no way to catch it.

Here’s a query — let’s call it Query A:

SELECT COUNT(*)
FROM   restaurants      r
JOIN   menu_items       mi ON r.restaurant_id = mi.restaurant_id
JOIN   sourcing_records sr ON mi.ingredient_category = sr.ingredient_category
JOIN   suppliers        s  ON sr.supplier_id = s.supplier_id
WHERE  r.guide_rating >= 1
  AND  s.supply_chain_type = 'industrial';

On its first run, the optimizer has no idea about the correlation, so the join explodes, producing tens of millions of rows before the filters bring it back down to almost nothing. On the next run, Optima Planning uses data from previous executions of the same query to produce a better plan:

But what happens if we have a similar query but distinct query? Here’s Query B. It joins the same three tables on the same columns as Query A, but the output columns are different, the rating cutoff is different, and it adds a sort:

SELECT r.restaurant_id, mi.menu_item_id, s.supplier_name
FROM   restaurants      r
JOIN   menu_items       mi ON r.restaurant_id = mi.restaurant_id
JOIN   sourcing_records sr ON mi.ingredient_category = sr.ingredient_category
JOIN   suppliers        s  ON sr.supplier_id = s.supplier_id
WHERE  r.guide_rating = 1
  AND  s.supply_chain_type = 'industrial'
ORDER BY r.restaurant_id, mi.menu_item_id;

The optimizer hasn’t seen this exact query before, so it struggles to produce a good plan, just like when we ran Query A for the first time. We end up with an exploding join and the query takes about ~13 seconds to execute.

With our new feature, we can apply the cardinality estimation information from query A to query B. This allows us to produce an improved plan when Query B is run as a brand new query for the very first time. The new plan executes in about half a second:

That’s roughly a 26x speedup on a query that had never run before and had no history of its own to draw on.

A production example

Here’s a real query from a large U.S.-based healthcare technology company that shows the power of this feature. A customer ran an analytics query with 19 joins and several correlated join predicates — the kind of shape where traditional cardinality estimation struggles. The optimizer generated a sub-optimal plan with an exploding join producing billions of rows. The query took almost two hours to complete.

Figure 4. Before applying cross-query-feedback: the optimizer underestimates a key join’s output leading to an exploding join.

On a subsequent query execution, the optimizer was able to apply cross-query feedback from a structurally similar but distinct query to improve cardinality estimation and produce a better plan, with an execution time of only 30 seconds!

What this example gets at is the real unlock of cross-query feedback: a query doesn’t need a track record of its own to benefit from one. If its join shape has shown up anywhere else in the workload, Optima Planning can put that experience to work immediately — even on a query’s very first run.

When this helps

This capability is most valuable when a genuinely new query shares a plan fragment or an individual operator with something that has already run. By matching inside the query, Optima Planning extends feedback to queries that are not the same. It helps most when:

• the query is expensive enough that a bad plan would trigger large scans or joins
• the query is genuinely new, a new join, a new table, a new combination, but extends an existing view, CTE or common join pattern already used elsewhere in the workload
• an ad hoc analyst query builds on a join pattern that recurs across many other reports, even though the specific report is new
• the SQL is freshly generated by BI tools or LLMs (for example, Cortex Analyst, Tableau Pulse)
• two BI tools or ORMs write the same join with a different join order — for now those are treated as different queries

These are the cases where matching on a plan fragment or an individual operator, instead of the whole query, can materially improve plan coverage.

How customers can observe Optima

Customers can monitor Snowflake Optima use in Query Profile under Query History, including the Query insights pane and Statistics pane, and through the QUERY_INSIGHTS view (full mechanism covered in “Grounding Query Plans in Real Data”).

For Optima Planning specifically, this means customers can see when Snowflake has applied optimization insights to improve plan quality. Today, this new matching shares the same customer-facing insight as Optima Planning’s other feedback: “This query benefits from Snowflake optima planning.” There is currently no separate message distinguishing a fragment or operator match from a match on the same query; both surface identically in Query Profile and QUERY_INSIGHTS.

Key takeaways

Optima Planning is about better evidence for better plans.Per-query feedback and cross-query feedback now cover more of the workload lifecycle:

• Recurring workloads: Dashboards, scheduled reports and ELT pipelines improve as Optima Planning learns about exploding joins from past executions.
• Queries that extend the known: A new join added on top of an existing view, CTE or common join pattern already used elsewhere now benefits from that fragment’s feedback, even on its own first run.
• Production impact: In a production workload, cross-query feedback reduced a 19-join analytics query from almost two hours to 30 seconds.

No manual tuning. No manual configuration. Just better plans.

*This post may contain forward-looking statements about future product capabilities. Actual results may differ. See our latest 10-Q.*

Snowflake Optima Planning: Cross-Query Performance was originally published in Snowflake Builders Blog: Data Engineers, App Developers, AI, & Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.