LLM Testing Is a Pyramid, and the Judge Belongs in the Middle
Put LLM-as-judge at the base and a judge sweep runs on every commit, pricing the gate out of existence in a quarter. Deterministic checks belong at the base, rubric eval in the middle, red-team and canary at the top.
A pull request runs a couple hundred judge calls across nine rubrics, takes four minutes, costs a few dollars, and goes green. Twelve hours later a customer reports the bot returning malformed JSON the app cannot parse. The judge scored 0.91 for groundedness and nobody asked whether the JSON parsed. A one-line schema check, costing nothing, would have caught it on the first PR. The expensive thing ran and the cheap thing did not.
That is the pyramid inverted. LLM testing is a pyramid like everything else, and putting the judge at the base is the anti-pattern that prices the gate out of existence. The shape is not new. The cost curve is: the cheapest checks still cost almost nothing, but the layer above them costs real money, so a team that runs a judge on every commit burns its budget on signal that does not compound while skipping the deterministic checks that catch the JSON failure, the missing citation, and the injection leak.
Get Future AGI’s stories in your inbox
Join Medium for free to get updates from this writer.
Five layers, bottom-heavy by volume and top-heavy by cost.
The five layers, briefly
- Deterministic, the base. JSON schema, regex on tool-call structure, exact-match on golden examples, citation-exists checks, and the sub-10ms scanners for jailbreaks and secrets. Runs on every commit with no baseline and no threshold tuning, because the check passes or it does not, and a failure short-circuits everything above it.
- Regression eval, the middle. Rubric scoring on a versioned golden set, gated on statistical significance, not a frozen number. Score against a trailing seven-day baseline and gate on the delta as well as a floor, so slow regressions and catastrophic drops both get caught. This is where the judge earns its bill and where most teams misspend it.
- Integration. Tool calls executed end to end, retrieval recall against ground-truth chunks, and multi-turn flows. Not adversarial, just checking the pieces fit, so it runs nightly rather than per PR.
- Red-team. Static injection suites of known patterns run pre-release, plus a generative red-team that attacks the current prompt weekly, with every successful attack promoted into the static suite.
- Canary, the top. Route 1 to 5 percent of live traffic to the new build, score it with the same rubrics CI uses, and auto-rollback on a per-rubric regression sustained past a pre-registered threshold. Pre-register the threshold before traffic flips, and alarm on a sustained delta rather than a single spike.
Wire it as three cadences
The PR gate runs the deterministic suite, then a cheap classifier-rubric subset, then the delta gate against the baseline, kept under a few minutes and a few cents. A nightly workflow runs the full rubric sweep plus integration and red-team against the full dataset and posts the day’s baseline. Canary lives in the gateway config, not in CI. The two anti-patterns to avoid are the inverted pyramid, with the judge at the base, and the flat stack, where every check is a pytest assertion that happens to call a model with no layers and no statistical gating.
The pyramid is a shape, not a vendor, and Future AGI ships the pieces so you build the shape rather than the plumbing. The ai-evaluation SDK, the open-source evaluation library, provides the Layer 2 rubric templates and the sub-10ms Layer 1 scanners, plus a CLI that ships the CI assertions and exit codes. traceAI, the open-source tracing layer, attaches the same rubrics to live spans, and the Agent Command Center, an OpenAI-compatible gateway you self-host, provides the shadow and canary routing for Layer 5. The rule that ties it together: the same rubric definition in CI and on the live span is the difference between an offline pass that ships and a 3am page.
What this comes down to
LLM testing is the same three-tier shape classical testing always used, with a cost curve that has flipped. Deterministic checks at the base on every commit for near nothing, rubric eval in the middle gated on significance, red-team and canary at the top where they are surgical and expensive. Put the judge at the base and the gate prices itself out inside a quarter. Put it in the middle, spend the cheap budget freely and the judge budget carefully, and the green check means something again.
