Three rounds of independent adversarial review broke our text-to-SQL guard. The third break has no patch, and it is the reason the agent in this series never writes a query at all.
This is a companion piece to Every Model Repeated the Fake Statistic, which covers how that agent keeps verified figures apart from unverified ones. This one covers the narrower question underneath it: if an agent needs to compute something from your data, how does it ask?
The short answer is that it does not ask in SQL. It sends a key, and trusted code turns that key into a query. What follows is why the obvious design fails, and what a concept file has to declare for the safe one to work.
The concept declares what may be asked
Three articles of OKF described facts — a concept file whose claim is a number, verified by recomputing it. The query tool needs something different: a concept describing a capability.
This is where the thesis could quietly die. If a capability concept is just markdown saying here’s the schema, here’s the syntax, then OKF has become a prompt file with YAML on top, and the one rule that made it OKF — verification executes, it does not lint — is gone.
So the rule holds; it just verifies a different claim:
type: capability
id: query_adult23
measures:
DIBINS_A:
label: "Currently takes insulin"
kind: prevalence
valid: [1, 2]
# A skip-pattern item is only asked of some people, so its VALID universes are part of
# its meaning. Without this list the tool computes the figure over "all adults", labels
# it that way, and is wrong by ~4.6x.
universes: [diagnosed_diabetes, diabetes_or_prediabetes]
DIBAGETC_A:
kind: mean
drop: [96, 97, 98, 99] # NOT 7/8/9 - those are real ages for this variable
universes: [diagnosed_diabetes]
# THE GATE. Each example is executed at compile time and must match.
examples:
- measure: DIBINS_A
universe: diagnosed_diabetes
expect: 31.96build.py runs those examples against the real slice. A capability whose example stops matching is quarantined — the agent loses the tool rather than gaining a broken one. The same rule that killed the wrong insulin number, pointed at a different kind of claim.
Two details in that YAML are worth pausing on, because both were bugs first.
universes exists because a real number with the wrong denominator is still wrong. Insulin use over "all adults" computes cleanly and reads 16.75%. It is meaningless — the question is only asked of people with diabetes. Binding each measure to the populations it is actually asked in makes the classic skip-pattern error inexpressible as a request.
drop is per-measure because 7, 8 and 9 mean different things in different columns. For a yes/no item they are refused / not ascertained / don't know. For age at diagnosis they are real ages. One blanket rule does both kinds of damage at once. It deletes the 18 people diagnosed at 7, 8 or 9 years old, because it reads their real ages as refusals. And it leaves 124 refusal codes in the average — three 97s and a hundred and twenty-one 99s — as if someone had been diagnosed at ninety-nine. The mean goes from the verified 47.41 to 49.27 — inflated by 1.85 years. Clean code, plausible answer, quietly wrong. That is the failure this project exists to catch, and it nearly shipped inside the fix.
The model never writes the query
The misconception: OKF can teach a model to query your database or call your API. Give it the schema, let it write the SQL, check the SQL before running it.
Get Andrew Wint’s stories in your inbox
Join Medium for free to get updates from this writer.
We built that. Three rounds of independent review broke it, and the third has no fix.
Round one. The guard required an aggregate somewhere in the query. An aggregate in a subquery satisfied it while the visible columns stayed raw — 200 complete respondent records in four calls, each with its survey weight.
Round two. The rule became “a plain column must have few possible values.” But FILTER and CASE WHEN select rows from inside an aggregate, so MIN(age) FILTER (WHERE weight = 3146.794) returns one person. And with no trick at all, the four "safe" columns combined into 35 groups — eight holding exactly one respondent, because the categories include rare refusal codes.
Round three ended it.
SELECT sum(DIBAGETC_A / (1 + (WTFA_A - 3146.794) * (WTFA_A - 3146.794) * 1e12)) FROM t
→ 61.00000000005 -- that person's exact age at diagnosisNo comparison. No CASE. No FILTER. Only + - * /, every column used as a rate calculation uses it. A steep enough fraction is a switch. Any language that can compute an average can pick out one person, so no checker reading query structure can tell them apart. The checker was right by its own rules. The rules were wrong.
So we removed the surface instead of defending it. The model sends keys, not SQL:
run_query(con, measure="DIBINS_A", universe="diagnosed_diabetes", group_by="SEX_A")Trusted code writes every SQL string, and the attack above is now an unrecognised universe key. The attacks are not blocked. They cannot be said.
That is what a capability concept actually does. It never teaches the model to query anything — it declares the finite set of questions that may be asked.
An independent review then listed every query the agent can run. There are 10. The smallest group any of them reports holds 1,523 people, against a floor of 30, and all three known attacks are unreachable.
Proof by listing everything, not by argument — and it works only because the list is short enough to write down.
What this cost, and what it bought
The obvious design — let the model write SQL, validate the SQL — is one an experienced engineer would defend, and we did defend it, through two rounds of review. The third round did not find a better filter to write. It found that no filter can exist, because the property being filtered for is not expressible in the query’s structure.
That is worth separating from ordinary hardening. Most security work is a list of things to block that grows as attackers get cleverer. This was the opposite: the list could not be completed in principle, so the surface had to go.
What it bought is a system whose safety can be demonstrated by listing everything it can do. Ten queries. Smallest reported group, 1,523 people. That is a claim a reader can check, which the previous design never allowed.
