If your dbt project’s semantic layer YAML still has semantic_models:, measures: and type_params:, it’s written in the legacy spec. dbt v1.12, dbt v2 and the dbt platform’s “Latest” release track support a new one, which dbt documents as the latest metrics spec. An autofix command does most of the conversion for you. This piece is about what changed and the parts the autofix leaves to you.

The idea behind the change

In the legacy spec, semantics lived in a separate top-level block that pointed at a model with ref(). Entities and dimensions were their own lists, each with an expr mapping back to a column, and every aggregation was declared twice: once as a measure, once as a metric.

The new spec turns that around. The semantic model is a property of the dbt model, and entities and dimensions sit on the columns they describe. dbt describes the result as “a 1:1 relationship between the physical columns and their semantic definitions.”

In practice that means one less structure to keep in sync. Column docs, tests and semantics live in the same YAML block, so renaming a column and updating its dimension happen in the same place.

Entities and dimensions, side by side

Here is the legacy version, from dbt’s migration docs:

semantic_models:
-name: orders
model: ref('orders')
entities:
-name: order
type: primary
expr: order_id
dimensions:
-name: ordered_at
type: time
type_params:
time_granularity: day

And the same thing in the latest spec:

models:
-name: orders
semantic_model:
enabled:true
agg_time_dimension: ordered_at
columns:
-name: order_id
entity:
type: primary
name: order
-name: ordered_at
granularity: day
dimension:
type: time

Three things disappeared: the ref(), the expr mapping (the column is the mapping now) and type_params.time_granularity, which becomes granularity on the column. agg_time_dimension moves from a defaults: block to the model level, and an individual metric can override it.

Measures are gone

This is the change most people notice first. The legacy spec needed a measure and a metric that pointed at it:

semantic_models:
-name: customers
model: ref('customers')
measures:
-name: lifetime_spend_pretax
agg: sum
metrics:
-name: lifetime_spend_pretax
type: simple
type_params:
measure: lifetime_spend_pretax

In the latest spec, a simple metric carries its own aggregation and lives on the model:

models:
-name: customers
semantic_model:
enabled:true
metrics:
-name: lifetime_spend_pretax
type: simple
agg: sum
expr: amount_pretax

type_params is flattened everywhere. Keys such as expr, percentile, percentile_type, non_additive_dimension, join_to_timespine and fill_nulls_with move up onto the metric itself.

Ratio, derived, cumulative and conversion metrics go in a top-level metrics: key. dbt says that top-level placement is required for any metric that depends on metrics or dimensions from a different semantic model. Several of those metric types also rename their inputs, which is easiest to see as a map:

The gotchas

Computed dimensions need derived_semantics

In the legacy spec, a dimension could be any SQL expression. In the new spec, a dimension defined on a column can’t take an expr. Computed dimensions go in a derived_semantics block that sits next to semantic_model: at the model level, as the dimensions docs show:

models:
-name: fct_transactions
semantic_model:
enabled:true
derived_semantics:
dimensions:
-name: is_bulk_transaction
type: categorical
expr:"case when quantity > 10 then true else false end"

If your legacy project leaned on case when dimensions, check where each one landed after the autofix.

Cumulative metrics need a metric, not a measure

type_params.measure becomes input_metric, and it has to reference a metric. With measures gone, a cumulative metric built on a measure needs a simple metric underneath it.

Conversion metrics changed shape

base_measure and conversion_measure become base_metric and conversion_metric. In dbt’s own before-and-after examples, constant_properties also goes from a map (plan: pro) to a list of base_property / conversion_property pairs. Read these by hand after migrating.

Cross-project references keep you on legacy

Per dbt’s docs, “cross-project references are not yet supported” in the latest spec, with support planned for a future release. If you reference models across projects, this is a blocker, not a gotcha.

Packages have to migrate first

The autofix tool only touches files in your current project, not dbt_packages/. If an installed package still defines metrics in the legacy spec, dbt may raise parsing or validation errors after you migrate. The documented order is: package maintainers run the autofix and release, then you upgrade the package, then you migrate.

The tooling is still catching up

dbt Copilot can’t yet generate semantic models in the latest spec. In the Studio IDE, the autofix can overwrite unsaved changes, so commit first. The IDE also validates YAML against the dbt v2 schema even if you run v1, and dbt’s advice there is blunt: “If the Studio IDE flags your YAML as invalid but dbt commands succeed, trust your run results.”

The migration, in order

The commands themselves are short:

dbt-autofix deprecations --semantic-layer
dbt parse
mf validate-configs   # on the dbt platform CLI: dbt sl validate

Note that dbt sl validate checks your local semantic manifest, not the one deployed on the platform.

One habit I’d add, though dbt doesn’t prescribe it: before migrating, run mf query --metrics <metric> --group-by metric_time__month --compile for your most important metrics and save the SQL. Run it again afterwards and diff. A migration that parses cleanly but changes a join or a filter is the one you most want to catch, and the MetricFlow commands make that cheap to check.

Should you migrate now? My rule of thumb: if neither blocker applies and you’re already on v1.12 or v2, do it soon. New docs and examples are written for the new spec, and help for the legacy one will only get harder to find. If a blocker does apply, waiting costs you little, because the legacy spec still works.

If you want a reference project already on the new spec, dbt Labs’ jaffle-shop uses it, though its dbt_project.yml requires dbt 2.0 or later, so adjust that line on v1.12.

Where Apache Ossie fits

The new spec isn’t the only way to define semantics in dbt any more. dbt v1.12 can also read Apache Ossie documents, the vendor-neutral format formerly called the Open Semantic Interchange, as an alternative to native YAML. Ossie entered the Apache Incubator in July 2026, and MetricFlow itself has been Apache 2.0 since October 2025.

The mechanics, from dbt’s docs:

• Ossie files are JSON, placed in an osi/ directory at the project root (or wherever osi-paths points).
• They must declare Ossie version 0.1.0 or 0.1.1. Anything else is a parse error.
• Ossie-sourced and native semantic models can coexist in one project.
• Every Ossie dataset must resolve to a dbt model in the same project. Sources, seeds, snapshots and external tables aren’t supported, and Ossie files inside installed packages are ignored.
• Unsupported metric types are dropped with warning I078, and parsing continues.
• v2 support is listed as “coming soon.”

That last behavior is the one to watch. A definition that silently disappears is worse than one that fails. If you experiment with Ossie, treat warning I078 as an error in CI. For now I’d keep authoring in native YAML and treat Ossie as an interchange format, which is what it was designed to be.

For the rest of the picture, including what’s free versus paid on the dbt platform and a full MetricFlow walkthrough, see this longer guide to the dbt Semantic Layer.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

The Complete Agent Stack

• Ontology: 1 Data Models · 2 Row-Level Security · 3 Data Plane · 4 Workspaces · 5 Self-Serve Onboarding ← You are here
• Agent harness: 6 Harness 2.0 · 7 MCP Connections
• Distribution: 8 BI Canvases · 9 Branded HTML Apps · 10 MCP for Claude · 11 Demo Projects
• Improvement loop: 12 Observability · 13 Golden Questions · 14 Builder Agent

The slowest part of building a data agent is the part nobody demos. Someone has to write a description for every table and column, a system prompt, and the first few skills. It takes weeks, and until it’s done the agent isn’t much use.

The last four parts made the case for a strong ontology. This one is about how to get a first version of it without spending the first month on it.

The cold start problem

Most teams hit three walls before they see a single good answer. Trying the product means booking a demo and waiting for an account. The workspace starts empty, so the first day goes on setup. And the descriptions, prompts and skills that make an agent accurate take weeks to write by hand.

None of that work is optional, because Parts 1 to 4 depend on it. But most of it is the same kind of work every time: read the tables, work out what they mean, and write it down.

Let an agent write the first draft

That’s the job we gave Dash, our setup agent. You connect your data, whether that’s a database, a CSV, the sample data, or any of 300 sources through the Data Plane. You can also connect MCP servers such as Notion or HubSpot and upload files that describe your business rules. Then Dash reads all of it: every table, every column, every MCP server and every file.

Depending on how much you give it, that takes from one minute to 30. Dash uses the same logic as the Builder Agent we’ll meet in Part 14, pointed at an empty workspace instead of a mature one.

“This just helps you set up your workspace from zero, so you don’t have to waste your time doing all this initial boilerplate setup.”

— Serguei Balanovich, CTO of Upsolve AI

What it produced in four minutes

In the demo, Dash ran against the sample connection and finished in about three and a half minutes. It wrote five table definitions and 26 column definitions. It added context files explaining how to read the table values and how to count Olympic medals, plus a file of assumptions and caveats it found while scanning.

It also noticed the connection held two unrelated datasets, Olympic results and New Zealand business financials. So it created two separate skills, one for each kind of question, and wrote a system prompt that tells the agent when to use which.

Let an agent write the first draft, and keep the review for yourself.

Everything lands in a draft

None of this goes live on its own. Dash writes everything into a draft of the workspace from Part 4, and you review the definitions, context files, skills and instructions before publishing. In the demo, publishing created version two, and the playground was ready to take questions straight away.

Give it real context

Dash can only describe what it can read. On sample data it makes sensible guesses, and it labels its assumptions so you know which ones to check. On your own data, the quality of the first draft depends on what you hand it: the Notion space where your metrics are defined, the doc that explains your pricing tiers, the file of business rules your analysts keep.

The more of your own documentation Dash reads, the less of its draft you’ll need to rewrite.

Free to try

The whole stack is available on the free tier at upsolve.ai. It comes with 2,000 one-time AI credits and every pro feature, limited only by the credits. That’s enough to build an agent, ask it real questions and put the answers on a canvas before you talk to anyone.

Where I’d start

Sign up, create a project, and connect the smallest real dataset you have, along with one document that explains it. Let Dash run, then read its assumptions and caveats file before anything else. That file tells you where your data is ambiguous, which is useful even if you never ship the agent.

The first draft of an agent used to take a month. Now the review is the slow part, which is where your time should go.

Read more about Self-Serve Onboarding in Upsolve →

For the step-by-step version, read the full guide on the Upsolve blog.

Next in the series: Part 6, Harness 2.0, on why we rebuilt our agent harness from scratch.

Prefer the 45-second version? Watch the Self-Serve Onboarding teaser on YouTube:

Follow me on Medium and LinkedIn for more on analytics agents, context engineering and production AI. If you want to try the stack yourself, start free at upsolve.ai.

The Complete Agent Stack

• Ontology: 1 Data Models · 2 Row-Level Security · 3 Data Plane · 4 Workspaces ← You are here · 5 Self-Serve Onboarding
• Agent harness: 6 Harness 2.0 · 7 MCP Connections
• Distribution: 8 BI Canvases · 9 Branded HTML Apps · 10 MCP for Claude · 11 Demo Projects
• Improvement loop: 12 Observability · 13 Golden Questions · 14 Builder Agent

Change one line in an agent’s system prompt and something else breaks: a skill that relied on the old wording, a golden question that now fails, a chart rule that no longer fits. The change was small. The trouble is that nobody could see everything it touched.

By this point in the series, an agent has a data model, permissions and sources. Over the next parts it will gain skills, memory, golden questions and prompts too. This part is about where all of that lives.

An agent is many files pretending to be one thing

In most setups, the pieces of an agent’s context live in different places. The data model sits in one tool, prompts in another, skills in a repo and examples in a spreadsheet. Each piece changes on its own schedule, so they drift apart.

The cost shows up in two places. Changing one piece means guessing how it affects the rest. Debugging means jumping between chat logs, traces and config files to piece together what the agent actually saw.

“Workspaces are pretty much everything we’ve been launching, all under a single pane of glass.”

— Serguei Balanovich, CTO of Upsolve AI

One pane of glass

An Upsolve workspace puts every part of the agent in one sidebar.

• Data model: the tables, descriptions and permissions from Parts 1 to 3.
• Knowledge base: the documentation the agent can read.
• Skills: such as generate SQL, spend details and spend driver.
• Assets: golden questions, charts and SQL examples for the questions the agent must answer.
• System prompts: including how to generate SQL and how to generate charts.
• Memory: an individual profile for every user who has used the agent.
• Chat history: who asked what, from which surface, and at what cost.
• Playground and deploy: where you test the agent and ship it.

If two pieces of context depend on each other, they belong in the same place.

Version the whole agent

The part I’d point a skeptic to is the versioning. The data model, knowledge base, skills and assets are versioned together as one release, with a production version and a draft. You choose which version to deploy, and it goes into your product as an iframe.

That changes how you make changes. You don’t edit a prompt in production and hope. You change the draft, test it, and promote the whole set at once. If something regresses, there’s one release to roll back to, instead of six files that each moved independently.

Version the agent as a whole, not its files one by one.

Test as anyone, with the trace beside you

The playground lets you ask questions as yourself or as any user in your organisation, which matters once row-level security is in place. As the answer comes back, a live trace runs alongside it: the golden examples it fetched, the schema it retrieved and the chart validation it ran.

In the demo, a question about monthly purchase order spend over the last 12 months comes back in seconds, with each step of the trace visible as it happens. You see the answer and the reasoning at the same time, before any user does.

It’s what makes self-improvement possible

There’s a second reason to keep everything in one place, and it’s the one I’m most excited about. When the chat history sits beside the context, an agent can read both. In Part 14, the Builder Agent uses exactly that to read how users struggle and propose fixes to the data model, prompts and skills. It can only do that because the workspace gives it the whole picture.

Where I’d start

Make your next change in a draft, however small. Test it in the playground as two different users and compare the traces. Then promote the draft as a new version and note what changed. After a few cycles, you’ll have something most agent teams don’t: a history of the agent that explains why it behaves the way it does.

An agent is only as consistent as the release it ships in.

Read more about Workspaces in Upsolve →

For the step-by-step version, read the full guide on the Upsolve blog.

Next in the series: Part 5, Self-Serve Onboarding, on getting a working workspace without a week of setup.

Prefer the 45-second version? Watch the Workspaces teaser on YouTube:

Follow me on Medium and LinkedIn for more on analytics agents, context engineering and production AI. If you want to try the stack yourself, start free at upsolve.ai.

Most claims about semantic layers and text-to-SQL accuracy trace back to a handful of studies. I went back to the ones that get cited most. They agree on the direction: giving a model written business context raises accuracy a lot, and more than switching models does. But they are small, mostly single-dataset, often run by a vendor, and they don’t test the same thing. In one study “semantic layer” means a compiled metrics engine. In another it means a 4 KB markdown file pasted into the prompt.

Here is what each study measured, what it didn’t, and how the numbers get stretched on their way into slide decks.

The seven studies, briefly

GSK’s pharmacovigilance study (JAMIA Open, 2025)

GSK researchers asked GPT-4 to turn 60 real questions from user logs into Oracle SQL against a drug-safety database. With the full schema (a 290-page PDF) fed through a retrieval pipeline, 8.3% of answers passed. Adding a business context document raised that to 78.3%, per the paper in JAMIA Open.

The detail most summaries skip is the third phase. Narrowing the schema to the 33 essential tables, without the context document, produced a 10% pass rate. Shrinking the schema barely helped. Explaining the business did.

The authors list the limits themselves: one model, one proprietary database, 60 questions. You will sometimes see this cited as “the Bayer study.” It’s GSK’s.

Cube’s paired benchmark (2026)

This one often appears twice in reading lists, as a Cube blog post and as arXiv 2604.25149. It is one study, written by Cube’s VP of Product and a co-author. It ran 100 questions on Contoso retail data in ClickHouse across Claude Opus 4.7, Claude Sonnet 4.6 and GPT-5.4, single-shot, with and without a 4 KB hand-written markdown document describing measures, conventions and disambiguation rules.

Accuracy rose 17 to 23 points on every model, to between 67.7% and 68.7%. The more interesting result: within each condition, the three models were statistically indistinguishable. The document explained the variance. The model choice didn’t.

What it did not test is a semantic layer engine. Nothing was compiled to SQL. It’s strong evidence for semantic content, which is a broader and different claim.

dbt Labs’ 2026 benchmark

dbt Labs reran its 2023 test on the ACME Insurance dataset: 11 questions, 20 runs each, with Claude Sonnet 4.6 and GPT-5.3 Codex.

• In scope: the dbt Semantic Layer scored 100% on both models. Text-to-SQL scored 62.5% and 51.2%.
• Questions needing too many entity hops: the semantic layer scored 0%, because it refused. Text-to-SQL scored 70.0% and 100.0% on those.
• All 11 questions: 72.7% for the semantic layer, 64.5% for text-to-SQL on both models.
• After an LLM added three dbt models: the semantic layer reached 98.2% and 100%. Text-to-SQL reached 90.0% and 84.1%.

The authors flag the main caveat themselves: to make text-to-SQL work they loaded the entire schema as context, “which isn’t practical for larger datasets.” It’s also 11 questions on one dataset, run by the vendor of the winning approach.

Spider 2.0 (ICLR 2025)

Spider 2.0 is 632 real enterprise text-to-SQL workflow tasks. In the November 2024 paper, an agent framework built on o1-preview solved 21.3% of them, against 91.2% on the older Spider 1.0 and 73.0% on BIRD.

That 21.3% is still quoted as if it described today’s models. It doesn’t. The top entry on the Spider 2.0-Snow leaderboard now scores 96.70. And Spider 2.0 was never a semantic layer study. It measured how hard raw enterprise schemas are, and the leaderboard since then shows how much agents that explore metadata and documentation can close that gap.

Sequeda, Allemang and Jacob (2023)

GPT-4, zero-shot, on an enterprise insurance schema: roughly 16% accuracy when asking the SQL database directly, roughly 54% when asking over a knowledge graph built with an ontology and mappings (arXiv 2311.07509). The insurance benchmark it introduced is the one dbt reused in 2026. It’s the origin of the “context triples accuracy” argument, and it predates several generations of models.

Snowflake’s semantic model study (2025)

On four BIRD subsets with Claude 3.5 Sonnet, agent-improved semantic models (relationships, descriptions, custom instructions) lifted average accuracy from 57% to 78%. Two caveats from the post itself: the comparison bundles the semantic model with Cortex Analyst’s agent system, so the gain can’t be credited to the model alone, and answers were graded by an LLM judge.

The DABStep ablation (2026)

This is the most useful study for separating causes. Qing Ye’s ablation held tools, retrieval instructions and the table allow-list fixed, then emptied the prose out of a YAML data contract. Semantic content lifted hard-task accuracy from 13.9% to 55.1% on the weakest of four models and from 37.0% to 77.4% on the strongest. Scaffolding alone added 0 to 5 points on two smaller models and 14 to 15 on two frontier models.

The contract also beat pasting the same knowledge into the prompt: generated SQL carried the fee rules up to 98% of the time, against 4% for the prompt version. Ungoverned runs issued 166 data-modifying statements. Governed runs issued none.

Where the studies agree

Context moves accuracy more than model choice. Cube found frontier models indistinguishable once they had the same document. GSK got from 8.3% to 78.3% without changing the model at all.

Meaning is the active ingredient, not schema size or plumbing. GSK’s trimmed schema barely moved. Ye’s scaffolding-only arm added far less than content. Even plain column descriptions help only a little on their own: an Alibaba study on BIRD found average accuracy across three open models went from 36.25% with no descriptions to 37.18% with generated ones and 38.59% with hand-written ones. Descriptions name things. Business context explains them.

Compiled layers change how you fail. The dbt semantic layer refused what it couldn’t model instead of guessing. In dbt’s words, “Text-to-SQL will cheerfully give you a wrong number.”

What none of them shows

A fair fight between compiled and written context. dbt compared its semantic layer against text-to-SQL with a raw schema. Cube compared a markdown document against a raw schema. Neither compared a compiled layer against an equally rich written document on the same questions. The closest is Ye’s study, where pre-computed views served as a ceiling, and the gap between the written contract and that ceiling shrank from 39 points to 5 as models got stronger.

Scale. The headline studies use 11, 60 and 100 questions, each on a single dataset.

Coverage in the wild. A compiled layer’s 100% applies to the questions it can answer. Nobody has published what share of a real company’s question queue falls in scope.

Upkeep. GSK notes that context documents need ongoing maintenance as schemas change. None of the studies measure what happens when the context goes stale.

How the numbers get misquoted

• “LLMs only solve 21% of enterprise SQL.” That’s a November 2024 baseline. The Spider 2.0-Snow leaderboard top score is now 96.70.
• “A semantic layer is 100% accurate.” On the in-scope questions of an 11-question set, after it refused the rest.
• “Two independent studies, from Cube and arXiv, agree.” They’re the same study.
• “The Bayer study.” The pharmacovigilance paper is GSK’s.
• “Descriptions over 20 words hurt accuracy.” In the Alibaba paper, the 20-word cap is a design choice motivated by that claim, not a measured result.
• “Semantics improve agent accuracy by 80%.” That’s a Gartner prediction for 2027 (“up to 80%”), not a measurement.

Questions to ask the next benchmark

New studies will keep arriving. These are the questions I now ask before quoting one:

1. How many questions, and were they taken from real logs or written for the test?
2. Who ran it, and do they sell the approach that won?
3. What exactly was “the semantic layer”: a compiled engine, a YAML file or a markdown document?
4. What did the baseline get: the full schema, retrieved snippets or nothing?
5. How were refusals scored?
6. Single-shot prompt, or an agent that could explore the data first?
7. Who graded the answers: execution match, a human or an LLM judge?

My reading of the evidence: write down business meaning first, because that’s what every study rewards, and compile the metrics that must be exact, because that’s what changes the failure mode. I go deeper on what goes into a semantic layer and on the context layer around it in two longer guides.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

A semantic layer answers one question very well: how is this metric calculated? That was the right question for dashboards. An AI agent asks it too, but it also needs to know what data exists, which values are valid, which answers are trusted and what this user is allowed to see. Most semantic layers were never designed to answer those.

My view: for AI agents, a semantic layer is necessary for the metrics that must be exact, and nowhere near sufficient for everything else an agent does. It’s worth being precise about which parts carry over and which don’t, because a lot of teams are budgeting for this right now. A Futurum survey of 818 enterprises found 44.5% plan to increase semantic layer spending over the next 24 months.

The problem the semantic layer solved

Ask finance, sales and product for last quarter’s revenue and you can get three defensible numbers: recognized net revenue by fiscal quarter, signed bookings by calendar quarter, ARR without usage overages. The semantic layer’s job was to write one of those down, with the SQL, the source table and an owner, so every dashboard computes it the same way.

It does that with a familiar set of parts: entities mapped to tables, dimensions to slice by, metrics, approved join paths and access rules, all compiled into SQL at query time. The consumer never writes the join or the formula, which is why two dashboards on the same layer can’t disagree. (I’ve written up how a semantic layer is put together in more detail elsewhere.)

But look at who sat at the other end. A human analyst opened the dashboard, picked a region from a dropdown and noticed when a number looked off. The BI semantic layer could leave a lot unsaid, because a person filled in the gaps.

What carries over to agents intact

Three parts of the semantic layer transfer to agents without loss, and arguably become more valuable.

Metric definitions. An agent answering questions in Slack all day has that many chances to pick a plausible definition that isn’t yours. A governed net_revenue removes the choice.

Join paths. Declared relationships prevent fan-out, where a one-to-many join silently double-counts. A person might notice a revenue figure that doubled overnight. An agent writing its own joins has no reference point, so the inflated number just looks like an answer.

Failing loudly. This one is underrated. In dbt Labs’ April 2026 benchmark, the dbt Semantic Layer answered 100% of in-scope questions correctly with both Claude Sonnet 4.6 and GPT-5.3 Codex, against 62.5% and 51.2% for text-to-SQL. On questions that needed more entity hops than the model covered, the semantic layer scored 0%, because it declined to answer. The authors put it this way:

“The Semantic Layer tells you it can’t answer. It never returns invalid data. Text-to-SQL will cheerfully give you a wrong number.”

A refusal costs a follow-up question. A confident wrong number can end up in a board deck.

What a dashboard never needed

This is where the BI-era design runs out. Each of the following was handled by a human or by the BI interface, and an agent has neither.

Allowed values

A dashboard filter is a dropdown. Nobody types 'Active' when the column holds 'active'. An agent writing a WHERE clause has no dropdown, so it needs the list of valid values before it writes SQL. Snowflake’s semantic view YAML spec added exactly this: sample_values, plus an is_enum: true flag that tells Cortex Analyst the list is complete.

What exists, and which table is authoritative

A BI model exposes a curated set of tables. An agent pointed at a warehouse sees fct_revenue and revenue_monthly_v2 and has no way to know which one the CFO trusts, or that one was abandoned in March. That’s structural knowledge, and most of it lives outside the semantic layer.

The long tail of questions

A semantic layer is exact on what you’ve modeled and silent on everything else. Dashboards only ask modeled questions, by construction. People asking an agent don’t know or care what’s modeled. In the dbt benchmark it took three additional dbt models before the semantic layer could answer every question in an 11-question set. Real question queues are a lot longer than 11.

Verified answers and enforced permissions

Row-level security carries over. But an agent also needs known-good answers to check itself against, golden questions to regression-test definitions, and permissions enforced outside the model rather than written into a prompt the model could reason around. Cube’s CEO Artyom Keydunov, arguing that the context layer needs a semantic layer, names the weakness of purely descriptive context like catalogs and glossaries: “nothing here produces an answer or enforces a rule.”

The companies that built BI semantic models have quietly conceded all of this. Microsoft’s Prep data for AI adds AI data schemas, verified answers and AI instructions on top of a Power BI semantic model, because when data is ambiguous, “AI systems can struggle to correctly interpret it.” You don’t bolt an AI layer onto something that was already enough.

Two camps, and why both are partly right

Teams are responding in two ways.

The first keeps the semantic layer at the center and wraps context around it. Keydunov calls the context layer “the superset” and the semantic layer “the one part that actually executes”: the piece that compiles certified metrics with access rules applied before any SQL reaches the warehouse.

The second skips the semantic layer. AngelList’s data team chose not to adopt a vendor semantic layer. Instead it keeps a markdown business glossary, a model catalog regenerated automatically from the dbt manifest (documenting some 27,000 columns) and a set of skill files, on the reasoning that an agent writes its own SQL and mostly needs legible, current text. Their line on maintenance is the best single sentence in this debate: “stale semantics are worse than none.” The trade-off they accept is that correctness depends on documentation quality and the model, not a deterministic planner.

I think the two camps are answering different questions. For the 10 to 20 metrics that go into board decks, you want something compiled that refuses rather than guesses. For the long tail, you want context the agent can read. dbt Labs reached the same conclusion from its own benchmark: “It’s both, for different things.”

The research also suggests the content matters more than the container. In a 2026 ablation study on the DABStep benchmark, adding semantic content lifted hard-task accuracy from 13.9% to 55.1% on the weakest of four models and from 37.0% to 77.4% on the strongest. Retrieval scaffolding on its own added between 0 and 15 points. What the agent is told matters more than how the telling is wired.

What to add on top of a semantic layer

If you’re putting an agent on a semantic layer you already have, these are the gaps to close. I sort them into three groups: structure, meaning and trust.

• Structure: register the tables the agent may use, with keys, and name the authoritative version when several look alike. List allowed values for low-cardinality columns and refresh them on a schedule.
• Meaning: your semantic layer’s metrics, plus vocabulary (“bookings” means signed contract value here) and the conventions an analyst would explain to a new hire.
• Trust: golden questions with verified SQL, run as evals whenever a definition changes, and permissions enforced deterministically, never delegated to the model.

If you don’t have a semantic layer yet, the same list is a starting point rather than a reason to wait. That’s the approach we took with Upsolve Data Models: tables and keys registered directly on the warehouse, definitions kept in a versioned system prompt, and selectable columns whose values are cached, with no semantic layer required. I’ve laid out how a context layer relates to the semantic layer in a longer guide.

The semantic layer was the right answer to the dashboard’s question. Agents ask more questions than dashboards did, and the semantic layer is still the best answer to one of them.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

Context engineering went from a phrase in a tweet to a Gartner recommendation in about six weeks in mid-2025. A year on, there’s a real body of research behind it: benchmarks on how models use long contexts, production data from teams running agents at scale, and surveys of where projects stall.

This piece collects the numbers I think matter most. Every figure links to its primary source, and where a number comes from a vendor testing its own product, I say so. I also run a company that builds analytics agents, so I have a stake in context mattering. I’ve tried to pick the numbers that would hold up even if I didn’t.

The short version

If you only have a minute, these six numbers carry most of the story.

• 44×: the jump in monthly US searches for “context engineering” in two months of 2025.
• 18 of 18: frontier models that got less reliable as their input grew, in Chroma’s context rot study.
• 11 of 13: long-context models that fell below half their short-context score at just 32,000 tokens.
• 55,000: tokens of tool definitions loaded before a conversation starts, in Anthropic’s example five-server MCP setup.
• 100:1: input tokens read per output token written by agents at Manus.
• 84%: fewer tokens used when stale tool results are cleared from an agent’s context automatically.

From a tweet to a discipline

The term had been around for a while, but it took off in a single week. In late June 2025, Shopify CEO Tobi Lütke posted that he preferred “context engineering” to “prompt engineering,” and Andrej Karpathy amplified it days later, describing the job as filling the context window with exactly the right information for the next step. (Simon Willison’s write-up collects both posts.) A month later, on July 28, Gartner published a note opening with a line that summed up the shift: “Context engineering is in, and prompt engineering is out.”

Search data shows how sudden it was. In Ahrefs’ US estimates, “context engineering” averaged about 115 searches a month from January to May 2025. It hit 1,602 in June and 5,072 in July, a 44-fold jump in two months. Over the past year it has averaged about 2,800 US searches and 14,000 global searches a month.

Two caveats keep this honest. Prompt engineering hasn’t gone anywhere in search: it still draws about 49,000 US searches a month, roughly 17 times as many. And Ahrefs’ figures for this term in 2026 look modeled rather than measured (its UK estimate sits at exactly 374 every month this year), so the chart covers the 2024–2025 breakout, where the data is clearly real. One more signal worth noting: “context rot” now gets about 1,200 US searches a month on its own. The problem has become as searchable as the discipline.

Bigger windows, same problem

Context windows now run to hundreds of thousands of tokens, and in some models millions. The research on how well models actually use that space is far less flattering.

• 18 out of 18. Chroma’s context rot study tested 18 frontier models, including GPT-4.1, Claude 4 and Gemini 2.5, and every one became less reliable as input length grew, even on simple tasks.
• 300 tokens beat 113,000. On the same conversational memory questions, every model family did significantly better with a focused prompt of about 300 tokens than with the full history of about 113,000 tokens containing the same answer.
• 11 of 13 below half. The NoLiMa benchmark removed word-for-word overlap between each question and the passage that answers it. At just 32,000 tokens, 11 of 13 models advertising windows of 128,000 or more fell below half their short-context score. GPT-4o, one of the strongest, dropped from 99.3% to 69.7%.
• The middle is a blind spot. Lost in the Middle found models use information at the start and end of a long context far better than information buried in between.
• One distractor is enough. Chroma found that a single passage that looks relevant but doesn’t answer the question measurably lowers accuracy.

Read together, these say an advertised context window is a ceiling, not a target. The NoLiMa result matters most for business data. People rarely ask questions in the same words their data uses (“revenue” versus a column called amt_net), and that’s exactly the condition under which long contexts fall apart.

Instructions and tools compete for the same attention

The system prompt and the tool list are part of the context too, and they grow quietly.

• 68% at 500 instructions. Distyl AI’s IFScale benchmark gave 20 frontier models up to 500 instructions at once. The best managed 68% accuracy at that density, and models increasingly favored whichever instructions came first.
• About 55,000 tokens before the conversation starts. In Anthropic’s example, a five-server MCP setup (GitHub, Slack, Sentry, Grafana and Splunk) loads 58 tools whose definitions take about 55,000 tokens. Internally, Anthropic has seen tool definitions reach 134,000 tokens before optimization.
• 49% to 74%. Loading tools on demand instead of all at once lifted Claude Opus 4’s accuracy on MCP evaluations from 49% to 74% (Opus 4.5 went from 79.5% to 88.1%), while using 85% fewer tokens.
• 72% to 90%. Adding a few example calls to tool definitions improved accuracy on complex parameters from 72% to 90%.
• 37% fewer tokens. Letting the model call tools from code, so intermediate results never enter the window, cut average usage from 43,588 to 27,297 tokens on complex research tasks.

Anthropic’s figures come from internal testing on its own models, so treat them as directional. But the pattern is consistent with the independent research above: the biggest wins come from what you keep out of the window. Anthropic names wrong tool selection as the most common failure, especially between near-identical tools, and that’s a context problem, not a reasoning one.

Agents run on input tokens

Chat is mostly short questions and long answers. Agents are the opposite, and the numbers show it.

• 4× and 15×. In Anthropic’s data, agents use about four times as many tokens as chat interactions, and multi-agent systems about 15 times as many.
• 80% of the variance. On the BrowseComp browsing benchmark, token usage alone explained 80% of the difference in performance, with the number of tool calls and the choice of model accounting for most of the rest.
• 90.2% better. A multi-agent setup, with Claude Opus 4 leading and Sonnet 4 subagents, beat single-agent Opus 4 by 90.2% on Anthropic’s internal research eval, largely by spreading the work across separate context windows.
• 100 to 1. At Manus, agents read about 100 input tokens for every token they write.
• 10× cheaper when cached. Manus pointed out that cached input on Claude Sonnet cost $0.30 per million tokens against $3 uncached, and called KV-cache hit rate the single most important metric for a production agent.

Put the 100:1 ratio next to the 10× cache discount and the economics come into focus. Almost all of an agent’s cost is reading context, so how that context is structured (what stays stable, what gets appended, what gets cut) sets the bill. That’s why context engineering became a cost discipline as much as an accuracy one.

Managing context pays off

If the problem is what’s in the window, the fix is deciding what goes in, what stays and what leaves. The early results are large.

• 84% fewer tokens. In a 100-turn web search evaluation, Anthropic’s context editing, which clears stale tool calls and results as the window fills, let agents finish workflows that otherwise failed by running out of context, while using 84% fewer tokens.
• 29% and 39%. On the same agentic evaluation, context editing alone improved performance by 29% over baseline. Combined with a file-based memory tool, the gain was 39%.
• +10.6% and +8.6%. The ACE framework (Agentic Context Engineering, presented at ICLR 2026) treats context as a playbook that improves from execution feedback. It beat strong baselines by 10.6% on agent benchmarks and 8.6% on finance tasks, and on the AppWorld leaderboard it matched the top-ranked production agent while using a smaller open-source model.
• 40% faster. When Anthropic had an agent repeatedly test a flawed tool and rewrite its description, later agents using the improved description finished tasks 40% faster.

The ACE paper also adds a useful warning. Repeatedly rewriting context into shorter summaries caused what the authors call context collapse, where details erode with every pass. Managing context isn’t the same as minimizing it. The goal is the smallest context that still holds what matters, and sometimes what matters is a long, detailed playbook.

Data agents: where context decides accuracy

Analytics agents make the clearest case, because a business question has a right answer you can check.

• 16% to 54%. In a 2023 benchmark from data.world’s AI lab, GPT-4 answered enterprise questions with 16% accuracy working from the raw database, and 54% once it had a knowledge graph of the business’s definitions. Same model, same questions. (data.world sells knowledge graph tooling, so weigh the exact figures accordingly.)
• 91.2% to 21.3%. Spider 2.0 moved text-to-SQL from tidy academic schemas to real enterprise databases, many with over 1,000 columns. An agent built on o1-preview solved 21.3% of tasks, against 91.2% on the original Spider and 73.0% on BIRD.
• 52.8% and 62.8%. When researchers audited two widely used text-to-SQL benchmarks, they found annotation errors in 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow. Rescoring 16 agents on corrected data changed their performance by up to 31% and moved some nine places on the leaderboard.

Together these frame the job. The models can write SQL. What they lack is the business knowledge that isn’t in the schema. And even the public scoreboards are noisy enough that a test set your own team has verified is the only one worth trusting.

The production gap

Adoption is real, and so is the gap between a working demo and a reliable system.

• More than 40%. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. It also estimates that only about 130 of the thousands of vendors selling “agentic AI” offer the real thing.
• 57% in production, 32% stuck on quality. LangChain’s State of Agent Engineering survey of more than 1,300 practitioners found 57% have agents in production, up from 51% a year earlier, rising to 67% at organizations with 10,000-plus employees. Quality was the top barrier for the second year running, named by 32%.
• 24% worse at saying “I don’t know.” Meta researchers tested 20 frontier models on underspecified or unanswerable questions and found that knowing when not to answer is still unsolved. Bigger models barely helped, and reasoning training cut abstention by 24% on average.

Gartner’s own explanation is telling: its analysts say current models lack the maturity to pursue complex goals or follow nuanced instructions over time. Holding instructions over a long task is as much a context problem as a model one, and “quality,” the top barrier in LangChain’s survey, is the umbrella for exactly the accuracy and consistency problems the research above measures.

The plumbing

The Model Context Protocol has become the standard way to connect agents to tools and data, and its growth explains why context budgets got tight so quickly.

• More than 10,000 servers. About a year after launch, there were over 10,000 active public MCP servers, covering everything from developer tools to Fortune 500 deployments, according to Anthropic.
• 97 million monthly downloads. The official SDKs passed 97 million monthly downloads across Python and TypeScript.
• A neutral home. Anthropic donated MCP to the Linux Foundation’s new Agentic AI Foundation, co-founded with Block and OpenAI and backed by Google, Microsoft, AWS, Cloudflare and Bloomberg.

Every one of those servers is also a potential stack of tool definitions in someone’s context window. Standard plumbing makes it easy to connect everything, and the tool numbers earlier show why loading everything at once is the wrong default.

What the numbers add up to

Taken together, the research points to five things worth keeping in mind.

The window is a ceiling, not a target. Every model tested degrades as context grows, and when questions and answers don’t share the same words, the drop starts early.

What you leave out matters more than what you add. The biggest measured gains, from on-demand tool loading, context editing and focused prompts, all come from keeping things out of the window.

The cost is in the reading. With agents reading about 100 tokens for every one they write, context structure sets the bill as much as the model price does.

Context beats model choice, but the model still matters. The same model went from 16% to 54% with better context. Yet Anthropic also found that upgrading to Claude Sonnet 4 helped more than doubling the token budget on Sonnet 3.7. Good context makes a good model better; it doesn’t replace one.

Measure on your own data. Two of the most widely used text-to-SQL benchmarks had errors in more than half their examples, and vendor figures come from vendors’ own tests. The number that matters is the one from your questions, with answers your team has checked.

If you want the techniques behind these numbers, I’ve written a longer guide to context engineering.

Context windows will keep getting bigger. The research says the job is filling them well.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

The Complete Agent Stack

• Ontology: 1 Data Models · 2 Row-Level Security · 3 Data Plane ← You are here · 4 Workspaces · 5 Self-Serve Onboarding
• Agent harness: 6 Harness 2.0 · 7 MCP Connections
• Distribution: 8 BI Canvases · 9 Branded HTML Apps · 10 MCP for Claude · 11 Demo Projects
• Improvement loop: 12 Observability · 13 Golden Questions · 14 Builder Agent

Connect an agent to Stripe, PostHog and HubSpot through three MCP servers, then ask how product usage relates to revenue. It calls one server, then the next, then the third, and tries to line the results up inside its own context window. Sometimes the answer is right.

Parts 1 and 2 assumed your data already sits in one warehouse. Plenty of teams aren’t there yet, and they still want an agent this quarter. This part is about how to give them one without asking the model to do arithmetic it was never built for.

Stitching MCP results together is where mistakes happen

MCP servers are a fast way to connect an agent to a tool. They’re well documented, and the agent usually knows how to call them. For analytics, though, they have three limits.

1. You only get what the provider exposed. If the MCP server doesn’t offer a table or a field, your agent can’t ask for it.
2. The join happens in the context window. The agent has to hold results from several sources and combine them in memory, which is exactly where models make things up.
3. Access is all or nothing. An MCP connection is either exposed to every user or requires each user to authenticate separately, and there’s no row-level security in between.

“It’s going to be very difficult for an agent to create an in-memory database where it can do proper joins between all these different sources, and it’s very prone to error.”

— Serguei Balanovich, CTO of Upsolve AI

One data model across many sources

The Data Plane connects hundreds of sources into a single Upsolve data model, with no warehouse required. That includes Stripe, PostHog, HubSpot, GitHub, Google and Facebook Ads, Google Sheets, CSVs and Instagram, along with BigQuery and ClickHouse if you want to join a warehouse to the apps around it.

Once the tables sit in one model, the agent writes real SQL that joins across them. That’s what makes multi-month and multi-year analysis possible, and it’s what makes the answer verifiable, because you can open the query and read the join.

Joins belong in SQL, where you can read them afterwards.

You choose what the agent sees

With an MCP server, the provider decides which data exists. With the Data Plane, you decide. You add the tables you need to the data model, then annotate and describe them the same way you would a warehouse table.

That matters more than it sounds. An agent works best on a small, well-described set of tables, and the best set for your questions is rarely the one a provider chose for everyone.

Expose the tables your questions need, and describe every one of them.

The same rules on every source

Because Data Plane sources live in the data model, the row-level security from Part 2 applies to them too. A global rule on a country or tenant column filters Stripe rows the same way it filters warehouse rows. You get one permission model across every source, instead of a different trust story for each connection.

Where MCP still fits

MCP servers are still the right tool for a different job. Reading a Notion page for context, creating a Linear ticket or saving an analysis somewhere your team works are actions rather than analytics, and that’s what Part 7 covers. The rule I use is simple: if the agent needs to count, sum or join it, it goes through the data model.

Where I’d start

Pick one question that needs two tools to answer, such as revenue by product usage tier, which needs Stripe and PostHog. Connect both through the Data Plane, add only the tables that question needs, and describe them. Then ask the question and read the SQL the agent writes. If you can follow the join, so can anyone who has to trust the answer.

Let the agent decide what to ask. Let the database do the arithmetic.

Read more about the Data Plane in Upsolve →

For the step-by-step version, read the full guide on the Upsolve blog.

Next in the series: Part 4, Workspaces, on keeping every piece of an agent’s context versioned together.

Prefer the 45-second version? Watch the Data Plane teaser on YouTube:

Follow me on Medium and LinkedIn for more on analytics agents, context engineering and production AI. If you want to try the stack yourself, start free at upsolve.ai.

Tags: context-engineering · ai-agent · data-agents · data-integration

I’ve spent three essays arguing that what goes into an AI agent’s context window matters more than which model reads it. I believe that. I also run a company whose product depends on it being true, which is a good reason not to take my word for it.

You don’t have to. Whether your own agent needs more context, less, or different context is a question you can answer on your own data, without a research team. The test is simple: same model, same questions, and only the context changes. Here’s how to set it up, what the research predicts you’ll see, and what to do with each result you might get.

The five setups

Hold everything constant except the context: the same model version and settings, the same system prompt, the same tools and output format. Then run every question through five setups.

1. Schema only. Table and column names with types, and nothing else. This is roughly what you get from a tool that points a model at your database and hopes.
2. Everything. The full schema, every metric definition, all the documentation you have, and every example query you’ve collected. As much as fits, on the theory that the model will find what it needs.
3. Curated. Only the tables relevant to the question, the governed definitions it touches, and a handful of vetted examples, assembled per question. If your product already builds context this way, this is your production setup, which is exactly why it needs testing.
4. Curated plus distractors. The curated context with near-duplicate definitions mixed in: an outdated version of each metric used, plus a few synthetic look-alikes. Write these yourself. Never use another customer’s real definitions or data, even in a test.
5. Curated, smaller model. The one setup where the model changes. Give the curated context to a smaller, cheaper model and see whether it keeps up.

The questions, and how to score them

The questions matter more than the setups. Use at least 60, ideally 100 or more, taken from what people actually asked your agent rather than questions you wrote yourself. Every one needs an answer your data team has verified. Mix four kinds: simple lookups, metrics with a company-specific definition, questions that join several tables, and deliberately ambiguous questions where the right behavior is to ask for clarification.

Don’t skip the verification, and don’t borrow answers from a public benchmark to save time. When researchers audited two of the most widely used text-to-SQL benchmarks, they found annotation errors in 52.8% of BIRD Mini-Dev and 62.8% of Spider 2.0-Snow: wrong reference answers, ambiguous questions and similar problems. Rescoring 16 agents on a corrected subset changed their performance by as much as 31% in relative terms and moved some up or down the leaderboard by nine places. If the answer key is wrong, your test measures the answer key.

Run each question three times per setup, because the same model can answer the same question differently. Compare setups on the same questions, and at this sample size treat a difference of a few points as noise rather than a winner.

Setting temperature to zero doesn’t fix this. Thinking Machines Lab sent the same prompt to a large open model 1,000 times at temperature zero, the setting meant to make output repeatable, and got 80 different completions. They matched word for word for the first 102 tokens, then split: 992 gave Queens as Feynman’s birthplace and 8 said New York City. The cause was how inference servers batch requests together as load changes, which you don’t control when you call a hosted model. In an analytics agent, a split like that can be the difference between two WHERE clauses. One run per question tells you what happened once.

Score results, not SQL. An answer is correct if its result matches the verified one, allowing for row order and rounding. For the ambiguous questions, it’s correct if the agent asks which definition you mean, or states the one it used and that one is right.

Then split every miss into two kinds. Either the agent said it couldn’t answer, or it returned a confident wrong number. That second kind is the one that costs you, because nobody notices it until a decision has been made on it. Track it separately, alongside input tokens, cost and latency for every question. And log the full assembled context for every run, so you can see why each answer came out the way it did.

Don’t expect a better model to bring that number down on its own. Researchers at Meta tested 20 frontier models on questions that were underspecified, built on false premises or simply unanswerable, and found that knowing when not to answer is still an unsolved problem, one that bigger models barely help with. Reasoning models did worse: their reasoning training cut abstention by 24% on average. A carefully written system prompt helped, but didn’t fix it. That makes the ambiguous questions the most important part of your set, and the confident-wrong count the number to watch most closely.

What the research predicts

Published research makes fairly specific predictions about how this should turn out, which gives you something to check your results against.

Schema only should come last. In a 2023 benchmark on enterprise SQL questions, GPT-4 scored 16% working from the raw database and 54% once it was given a knowledge graph of the business’s definitions. A bare schema leaves the model guessing what every metric means.

Everything should lose to curated, despite containing strictly more information. Chroma’s context rot study gave models the same questions with either a focused prompt of about 300 tokens or a full history of about 113,000 tokens containing the same answer. Every model family did significantly better on the focused version.

Distractors should cost accuracy. The same study found that even one passage that looks relevant but doesn’t answer the question lowered performance. It also found model families fail differently: some tend to abstain when uncertain, others answer confidently and wrongly. That second pattern is exactly what the confident-wrong count is there to catch.

Position should matter too. The Lost in the Middle study found that models use information best when it sits at the start or end of a long context, and noticeably worse when it’s buried in the middle, even models built for long inputs. That suggests a cheap extra run for the everything setup: once with the relevant definitions near the top, once with them in the middle. If accuracy moves, your production context has an ordering problem as well as a size problem.

The Chroma study turned up something stranger. Across all 18 models, the answer was found more reliably when the surrounding text was shuffled into nonsense than when it read as coherent documents. The researchers don’t claim to know why. That’s a useful reminder that intuition about what a model will find easy is unreliable, which is the whole case for testing.

The last prediction is the least settled, and the one I’d most like to see tested widely: a smaller model with curated context should beat a larger model working from the schema alone. If context really matters more than the model, that’s the result it implies, and it would change how most teams think about cost.

How to read your results

Each pattern you might see points to a different next step.

• Curated beats everything by a wide margin. Your agent is paying for context that makes it worse. Start trimming: cap what retrieval returns, drop stale examples, and audit the system prompt for rules nobody can explain.
• Everything roughly matches curated. Either your model handles long context well at your sizes, or your “everything” isn’t big yet. If accuracy is close, compare cost, because the cheaper setup wins a tie. Then rerun the test as your data grows.
• Distractors hurt, mostly as confident wrong answers. That’s the dangerous pattern. Find where look-alike content gets into real contexts, such as old metric versions, duplicated docs or examples copied between customers, and remove it or version it.
• Schema only isn’t far behind. Your questions may be simpler than you think, or your schema unusually well named. Look at the company-specific metric questions on their own before concluding that context doesn’t matter for you.
• The smaller model keeps up. You may be paying for a bigger model than your context requires. Before switching, compare confident-wrong rates, not just accuracy.

Mistakes that make the numbers lie

A test like this is easy to run badly, and a bad run is worse than none because it looks like evidence.

• Writing the questions yourself. Your questions match how you think about the data, not how users ask about it. Pull them from real logs.
• Grading with the setup visible. If people check any answers by hand, hide which setup produced them. Everyone has a favorite.
• Distractors that are too easy. A look-alike definition that’s obviously wrong tests nothing. The useful ones are close enough that a tired analyst might mix them up.
• Reporting only accuracy. Two setups with the same accuracy can differ a lot in confident wrong answers and cost, and those are often the numbers that decide.
• Treating one result as general. One deployment and one model family tell you about your agent, not every agent. If you change models or your data grows a lot, run it again.

Start with the log

You’ll probably learn something before you score a single answer. The test requires logging the full context for every question, which is worth doing anyway. Most teams that do it for the first time are surprised by what’s in there.

If you want the techniques behind a curated context, I’ve written a longer guide to context engineering.

Everyone has an opinion about how much context an agent needs. A week of testing gets you an answer.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

Every AI analytics demo looks the same. Someone types a question in plain English, a chart appears a few seconds later, and the number is right. Watch five of them in a week and you’ll struggle to remember which was which.

That’s because a demo shows you the one part of these products that has become easy: a capable model turning a question into SQL. It hides the part that decides whether the tool will work on your data, which is everything the model is given to work with.

So the most useful question I know to ask a vendor isn’t about the model at all. It’s this: when a user asks a question, what exactly goes into the model’s context window, and how is it scoped? In my experience, the quality of the answer predicts the quality of the product better than any demo does.

A disclosure before going further: I’m the co-founder and CEO of Upsolve, which sells one of these tools. I’ve tried to write the questions below so they’re fair to every vendor, including us, and there’s a section near the end on where our own answer used to fall short.

Why the demo can’t tell you

A demo runs on the vendor’s terms. The schema is small and clean, the questions have been tried before, and there’s one company’s worth of data with one set of definitions. None of that is dishonest. It just removes everything that makes your environment hard.

What’s left is the model, and the model is the part that matters least. In a 2023 benchmark from data.world’s AI lab, GPT-4 answered enterprise questions over a raw insurance database with 16% accuracy. The same model, given a knowledge graph that encoded the business’s definitions and relationships, reached 54%. Same model, same questions, more than three times the accuracy, and the only thing that changed was the context.

Keep in mind that data.world sells knowledge graph tooling, so weigh the exact numbers accordingly. And models have improved a lot since 2023. But the lesson for buyers hasn’t changed: when two tools run on similar models, the difference between them is the context.

That’s why questions about the model (“Which LLM do you use?”, “Can we bring our own?”) tell you so little. Many vendors can swap models quickly. What they can’t swap quickly is the machinery that decides what the model sees.

Six questions to ask instead

Here are the questions I’d bring to any vendor, each aimed at a different part of what the model sees. For each one, here’s what a good answer and a red flag tend to sound like.

1. What goes into the context window when a user asks a question?

A good answer is a demonstration, not a description. They open a real assembled context for a sample question and walk you through it: the instructions, the definitions and schema that were retrieved, any examples, and the query that came out. A red flag is the conversation drifting to which model they use, or “our AI handles that.”

2. Where do metric definitions come from, and who owns them?

A good answer: definitions live somewhere you can see and edit, they’re versioned, a change takes effect everywhere at once, and the model selects from them rather than inventing them. A red flag: “the AI learns what churn means from your schema.” Your schema doesn’t know what churn means. Your finance team does.

3. How are permissions enforced, and is the model trusted with any of it?

A good answer puts enforcement below the model, in scoped credentials and row-level rules applied before results come back. The context only tells the model what the user can see. A red flag: “we instruct the model to only show data the user has access to.” An instruction is a request, not a control.

4. What else lands in the window, and where does it come from?

A good answer lists it without hesitation: example queries, sample column values, memory, cached answers, retrieved documents. Then it explains how each is scoped to your company and your user. A red flag is a pause followed by “it’s all secure.” [Optional: link to the second article here once it’s live, where it covers how this context leaks between customers.]

5. Can I see what the model saw for any answer?

A good answer: every answer links to a trace of the assembled context and the query it ran, and your admins can open it, not just the vendor’s engineers. A red flag: you can see the SQL but not why it was written that way.

6. How do you measure accuracy, and on whose questions?

A good answer: they’ll run your own known-answer questions on your data during the trial, at realistic context sizes, and keep running them after you go live. A red flag is a single headline accuracy number from a public benchmark.

Four tests to run in the trial

Answers on a sales call are still answers on the vendor’s terms. A trial is where you check them, and these four tests take an afternoon:

1. Ask an ambiguous metric question. Pick a metric your company defines in a specific way, like churn or active customer, and ask for it without explaining. A good tool either asks which definition you mean or tells you which one it used. A weak one silently picks something plausible.
2. Ask questions you already know the answer to. Pull ten numbers from reports your finance or ops team trusts and ask for each one. Count how many match, and for the ones that don’t, open the trace and see why.
3. Ask for something you shouldn’t see. Log in as a restricted user and ask, by name, for data outside that user’s scope. The right result is that the data never comes back from the database, not a polite refusal from a model that had it in front of it.
4. Change a definition mid-trial. Update one metric’s definition and ask the same question again. The answer should change everywhere at once, without anyone re-prompting anything.

For the answers that matter most, compare the answer with its trace. If you can’t get a trace, that tells you something too.

Where our own answer fell short

I’d be suspicious of a buyer’s checklist that its author’s product passes perfectly, so here’s ours, honestly.

Every team building these tools is somewhere on this path. What matters is whether a vendor can tell you where they are on it, and show you.

What you’re actually buying

When you buy an AI analytics tool, the model is the part you could get almost anywhere. What you’re really buying is the system that decides, for every question, which definitions, permissions and history the model gets to see. That system is invisible in a demo and decisive in production.

So ask to see it, and keep asking until someone opens a real trace in front of you. If you want the vocabulary for the follow-up questions, I’ve written a longer guide to context engineering.

If they can’t show you what the model saw, they can’t tell you why it was right.

Follow me on Medium and LinkedIn for more on analytics agents, context engineering, and the practical infrastructure behind production-grade AI.

The Complete Agent Stack

• Ontology: 1 Data Models · 2 Row-Level Security ← You are here · 3 Data Plane · 4 Workspaces · 5 Self-Serve Onboarding
• Agent harness: 6 Harness 2.0 · 7 MCP Connections
• Distribution: 8 BI Canvases · 9 Branded HTML Apps · 10 MCP for Claude · 11 Demo Projects
• Improvement loop: 12 Observability · 13 Golden Questions · 14 Builder Agent

Every security review of a data agent comes down to one question: what stops it from showing customer A’s data to customer B? If the honest answer is “we told it not to”, the review is over.

In Part 1 we gave the agent one definition of every metric. This part is about the other half of the ontology, which decides what each user is allowed to see.

A permission written in a prompt is a suggestion

The easy way to secure an agent is to describe the rules in its instructions: only show this user their own region, never return another tenant’s rows. It works in the demo. In production, an agent that enforces its own permissions can be confused by an odd question, manipulated by a prompt injection, or simply wrong.

“Row-level security is always the number one concern our customers have when they deploy an agent against their data and ship it into production.”

— Serguei Balanovich, CTO of Upsolve AI

That concern is why so many AI projects stall at the security review. A multi-tenant SaaS company can’t put an agent in front of customers without hard data isolation, and nobody can sign off on isolation that depends on a model behaving.

Take security out of the agent entirely

The fix we chose is to make permissions deterministic and keep them away from the agent. In Upsolve, row-level security runs on the server as a pre-filter, before any query reaches your database. It applies to every query, whether the agent wrote it or a person typed it.

The agent never even sees the rules. That’s deliberate, because it keeps a clean line between the agentic part of the system, which should learn and adapt, and the deterministic part, which should never change its mind.

Anything that must always be true shouldn’t depend on a model.

Rules come from who the user is

The rules are built from properties. You register organisations and users in Upsolve and give them properties such as country, tenant or role. In our procurement demo, the IT organisation has the property country Japan, and each user carries properties of their own.

Then you write rules against those properties. A global rule like column country = user.country applies to every table in the data model that has a country column, and Upsolve shows you which tables it will touch. Some tables need nothing: the demo’s contracts table has no country column, so no rule applies, which is fine by design. When a rule needs a join into another table, you write custom SQL for that table instead.

Each rule is optional or required. If it’s optional and a user has no country property, the rule is skipped. If it’s required, that user sees no data at all for anything the rule covers.

Decide what a missing property means before a user finds out for you.

Test it as anyone

Rules you can’t inspect are hard to trust, so every table lets you preview a query as any user. In the demo, Grace Tan’s query comes back with a filter for India, Australia, Singapore and Japan, and you can read the exact WHERE clause Upsolve adds. Switch to Carlos Ruiz, who is based in Brazil, and the same query returns a single supplier from São Paulo.

That preview is also what you show a security reviewer. It turns “trust us” into a filter they can read.

If you can’t show a reviewer the WHERE clause, you don’t have row-level security.

One agent, every tenant

Once isolation is deterministic, the deployment model changes. You don’t need an agent per customer or a copy of the data per tenant. You deploy one agent with one data model, and thousands of customers each see only their own rows.

It also makes the rest of the stack safer. Later in the series, agents will learn new skills, update their memory and rewrite their own context. None of that can loosen a permission, because none of it touches the rules.

Where I’d start

List the tables that hold tenant or customer data, and find the column in each that identifies who owns a row. Write one global rule against that column, make it required, and preview the same query as two users from different tenants. If the two WHERE clauses look right to you, they’ll look right to your security team.

Let the agent be clever about everything else. Who sees what isn’t a place for clever.

Read more about Row-Level Security in Upsolve →

For the step-by-step version, read the full guide on the Upsolve blog.

Next in the series: Part 3, the Data Plane, on why your agent shouldn’t be joining data inside its context window.

Prefer the 45-second version? Watch the Row-Level Security teaser on YouTube:

Follow me on Medium and LinkedIn for more on analytics agents, context engineering and production AI. If you want to try the stack yourself, start free at upsolve.ai.