Route First, Retrieve Second

The question that comes after every article

Two articles ago I argued that questions have a shape, and that the shape decides whether search, a graph, or an agent over both can answer them. The follow-up put a rule of thumb on trial and found the cell nobody owns. Both times, the same question came back from readers: fine, so which one do I build?

My answer has not changed. All three. And something that decides.

This article is about the something. It is the last piece of this series, and the one I have come to believe matters more than any of the three designs behind it, because it is the only place where the economics of the whole system are decided. A router that reads each question and sends it to the cheapest design that can answer it is the difference between paying for one design and paying for three. It is also, as it turns out, the place where a system quietly fails when the router is confident and wrong.

Everything below was run on the same invented payments company as before, on Microsoft Azure AI Foundry, with Azure AI Search holding the pages and Neo4j holding the graph. Fifty questions, three router models, every routing decision recorded, and for the 32 questions that have a computed truth, every outcome graded against it.

Two routers, two different decisions

The word “router” now means two things in this stack, and they should not be confused.

Press enter or click to view image in full size

The question router decides which design runs: Classic RAG, GraphRAG, or the agentic hybrid. It is a model call you write and own. It knows your stores, your ontology and your prices, because you tell it.

The model router decides which model serves a given call. On Azure it is a deployment that fronts a pool of models and picks one per request from the prompt’s difficulty, its tools and its length. It is the platform’s, you configure its mode, and it knows nothing about your data.

They sit in sequence. The first decides who answers; the second decides how expensive a brain the answerer gets. Most of this article is about the first, because that is where the architecture lives. The second gets its own section, with measurements, near the end.

How the question router works

It is one call to a small model with strict structured output. The prompt is 806 tokens: the six shapes from the first article with the design each maps to, a short description of the three designs, nine rules of thumb, and eight generic examples, none of which is a test question. The schema is the whole contract:

SCHEMA = {
    "type": "object",
    "properties": {
        "shape": {"type": "string", "enum": ["page", "chain", "set", "absence", "definition", "mixed"]},
        "route": {"type": "string", "enum": ["rag", "graph", "hybrid"]},
        "confidence": {"type": "number", "description": "0 to 1, how sure you are of the route"},
        "reasoning": {"type": "string", "description": "one sentence, at most 30 words"},
    },
    "required": ["shape", "route", "confidence", "reasoning"],
    "additionalProperties": False,
}
fallback = decision["confidence"] < 0.6 and decision["route"] != "hybrid"
route = "hybrid" if fallback else decision["route"]

The model cannot invent a fourth route or skip the confidence. Below a confidence of 0.6, the question goes to the hybrid, which can answer anything at a price. On gpt-4o-mini, the demo’s router, a decision takes about a second and costs 0.016 cents. On gpt-4.1 it takes 1.4 seconds and 0.22 cents, more than half of what Classic RAG charges to answer the question. I will come back to what the confidence is actually worth.

The experiment

Three question sets.

• The 32 answered questions. The twelve from the first article and the twenty from the second. For every one, I have the recorded verdict and cost of all three designs on gpt-4.1. That lets me replay any routing strategy without running a model: route the question, look up what the chosen design did and what it cost, add the router’s own cost.
• 18 new questions written to fool the vocabulary. “How many minutes did the double-charge outage last?” sounds like a count; it is one number on one page. “Give me the full list of steps to fail over to the backup acquirer” sounds like a set; it is a procedure. “Compare the recovery point objective of payments-db and ledger-db” sounds like analysis; it is two sentences on two pages. These have no computed answers and were never run through the designs; they test only whether the router reads shape or keywords. Each has an expected route decided by where the answer lives in the data.
• The same 20 matrix questions again, with the router given one extra thing: a compact description of what the graph actually holds.

Three router models: gpt-4o-mini, gpt-4.1 and gpt-5-mini. Strategies compared against them: always one design, all three at once, and an oracle that picks the cheapest design that answered correctly, which is the ceiling any router can reach.

What routing is worth

Start with the ends of the scale. Running all three designs on every question and keeping the best answer scores 92% and costs 1.99 cents a question. The oracle scores the same 92% for 0.58 cents, because it never pays for a design that was not needed. That gap, 1.41 cents on every question, is what a routing decision is worth at its best. Always running the hybrid, the design that can do everything, lands at 91% and 0.97 cents: most of the saving, not all of it, and one point short on accuracy.

Now the routers. All three models score 83% and send roughly two thirds of the questions to the graph. gpt-4o-mini does it for 0.64 cents a question, gpt-5-mini for 0.63, gpt-4.1 for 0.81, where the difference is almost entirely the router’s own price. So a router costing a hundredth of a cent captured most of the oracle’s saving. It also lost eight points of accuracy against always-hybrid, and I want to be precise about where.

Split the 32 by origin and the picture sharpens. On the first twelve questions, every router scores 100%, the same as always-hybrid, and the gpt-4o-mini router does it for 0.94 cents against the hybrid’s 1.26, a quarter less. On the twenty matrix questions, the routers score 72.5% against the hybrid’s 85%. Every point of that difference comes from one cell: text, worked out. The trap from the second article. The router reads “which run books contain a rollback procedure” as a set, which it is, and sends it to the graph, which is right in shape and wrong in fact, because procedures were never loaded into the graph. The graph agent answers that no run books mention rollback. The router did its job. The data had not been given the shape the router assumed.

A router can only choose among the cells that have an owner. Where no design answers, the router cannot help, and it will not know that it did not help.

Over-routing and under-routing

gpt-4o-mini differed from the oracle on ten of the 32 questions. Seven of those were over-routes: the chosen design answered correctly but a cheaper one would have too. Four lookups about ownership, on-call and dependencies went to the graph at 0.42 cents instead of search at 0.31, because the router’s rules say ownership is structure, and they are not wrong. Three questions went to the hybrid when a cheaper design would have done. Over-routing is the cheap mistake: you pay a little more and you get the answer.

Three misroutes cost correctness. Two sit in the trap cell: text-only questions sent to a graph that had nothing to query. The third sent a lookup to the graph and got a partial, because the question said “services” and the graph returned a data store as well, which the judge did not forgive.

That asymmetry is the first design rule I would carry out of this: when in doubt, over-route. A wrong cheap answer is the expensive one, because somebody acts on it. The hybrid is the right default for doubt. The problem, as the next section shows, is that the router is never in doubt.

The words lie, and the router mostly sees through them

The 18 routing questions were written to separate reading shape from matching keywords. By the strict count, where only the cheapest correct design is a hit, gpt-4.1 and gpt-5-mini routed 16 of 18 as expected and gpt-4o-mini 13 of 18. Two of the labels are judgement calls, and the table marks them: the on-call lookup, which the graph answers as well as search does, and the two-part question about personal data, which the hybrid answers as well as the graph does. Count those as hits and the two larger models are 18 of 18. I report the strict number because it is the one I would have to defend.

The successes are the interesting part. “How many minutes did the double-charge outage last?” went to search on all three models, despite “how many”. “Give me the full list of steps to fail over to the backup acquirer” went to search, despite “full list”. “What changed after the payments database migration outage, and is anything still pending?” went to search on two of three, despite “pending” sounding like a status query. A router that reads the question as a whole handles the vocabulary that trips keyword rules.

The failures are instructive too. “Compare the recovery point objective of payments-db and ledger-db” went to the graph on all three models, with confidence of 0.90 to 0.95. The RPO is a sentence on each store’s page; the graph, before the second article’s extension, did not hold it. The word “compare” won. And “which data stores hold personal data, and which services write to them” went to the hybrid on two models, an over-route: both halves are in the graph, but a question with two parts reads as “mixed”. Over-routing again, and again the safe direction.

The confidence that means nothing

In the first article I wrote that over-routing costs money, under-routing costs correctness, and that the confidence threshold “is the dial between them”. I measured the dial. It is not connected to anything.

Every miss came with high confidence. gpt-4.1 reported 0.95 or more on all 32 decisions, including the six where the routed design did not answer correctly. gpt-4o-mini never went below 0.70 on any of its fifty decisions, and its three misses that cost correctness all carried 0.90. gpt-5-mini’s lowest was 0.86. The fallback to the hybrid, which triggers below 0.6, fired zero times in 150 decisions across the three models. The router is not calibrated; it is confident.

The sweep makes it plain. Raise the threshold from 0 to 0.85 and the score does not move. Raise it past the model’s habitual confidence and every question falls through to the hybrid, and you are paying always-hybrid prices with a router in front. gpt-5-mini, whose confidence spread is slightly wider, catches one miss at a threshold of 0.9. That is the whole benefit, on this data, of asking the model how sure it is.

I do not think this is a property of these three models. A model asked to grade its own decision in the same breath as making it will mostly agree with itself. If you want a signal, you need one the router cannot generate.

The signal the router cannot see, and the design can

There is a better signal, and it was in the recorded runs all along. On four of the five trap questions, the graph agent never ran a query. It read the ontology, saw that action items, procedures, retention and recovery objectives were not there, and said so. On the fifth it ran one query and got zero rows. The design knew it had nothing. The router did not.

So I replayed one more strategy over the recorded runs: route as before, and if the chosen design is the graph and the agent either gives up without a query or gets only empty results, escalate to the hybrid and pay for both. On gpt-4o-mini this fires on exactly the five trap questions and nowhere else. The score goes from 83% to 91%, the same as always-hybrid, at 0.75 cents against the hybrid’s 0.97. A quarter cheaper, for the same accuracy, with a rule that costs nothing to implement: a count of rows.

That is the second design rule. Do not fall back on the router’s confidence. Fall back on the design’s result. An empty graph result, a search whose top passages do not mention the entity asked about, an agent that answers without calling a tool: these are observable, and they are honest in a way a self-reported number is not.

The router that knows what the graph holds

The trap cell exposed a deeper limit. The router reads the question. It does not know the data. “Which run books contain a rollback procedure” is a set, and the router is right to say so; the mistake is assuming the set exists somewhere queryable. So I gave the router one more thing: a compact list of what the graph actually holds, one line per class with its properties and relationships, 242 tokens, and one rule: if a question needs something complete or counted that the graph does not hold, send it to the hybrid.

Figure 6 is one question, end to end, from the records. Without the hint, gpt-4.1 routed the rollback question to the graph at 0.95, and the graph agent answered that no run books mention it. With the hint, the same model routed it to the hybrid at 0.80, with the reason “this information is only in document prose, not the graph”, and the hybrid searched the run books and named both. After the structure from the second article was loaded and the hint updated to say so, the router sent it back to the graph at 0.90, “a graph query over Run book and Procedure”, and the graph agent answered with one query.

Across the set, the hint took all three routers from 72.5% to 82.5% before the structure existed, by moving three to five of the trap questions away from the graph. It did not reach always-hybrid’s 85%, because the hybrid itself only answered half the trap questions, and because gpt-4o-mini used the new rule to over-route a counting question that the graph could have answered. A router that knows the data is a better router. It is not a substitute for the data having the right shape.

After the structure was built, the same routers, told nothing new, scored 97.5% on the matrix set, against 95% for always-hybrid, at 0.60 cents against 0.77. With the owner of every cell in place, a hundredth-of-a-cent router beat the design that can do everything, on both score and price. That is the sentence I wanted from this whole series, and I did not get to write it until the trap was closed.

The other router: letting the platform pick the model

The model router is a different animal, and it is worth being precise about what it is. On Azure AI Foundry it is a deployment that fronts a pool of models. Every call to it is classified and handed to one of them, in one of three modes: cost, balanced or quality. You never name the model. The response tells you which one answered, and nothing else. I ran all three designs on the twenty matrix questions through each mode, the same questions and judge as before, so the fixed-model numbers from the earlier articles are directly comparable.

Four things came out of it.

The pool is not yours, and it moved under me. When I first wired the model router into this setup, a week before the previous article’s runs, the pool served three models. By the time of these runs it served nine, including three from a newer generation and one that is not an OpenAI model at all. In cost mode, the small models took almost every call. In quality mode, the graph and hybrid agents were served by the same frontier model on every single call. In balanced mode, the router escalated between a third and a half of the agents’ calls to a frontier model, and it was the mode where the hybrid scored lowest, 87.5%, partly because one question, about what breaks if the event stream fails, was refused outright by a content filter on the reasoning model that happened to be chosen. A question about an outage, flagged as a policy violation, by a model I did not pick. That is the price of not picking.

Cheap and good are not opposites here. Cost mode’s hybrid scored 92.5%, higher than gpt-4.1’s 85% on the same questions, for 0.19 cents a question against 0.80. The small models in the pool read documents far more willingly than gpt-4.1 did, nineteen full document reads across the five trap questions against none, which is why they did better in that cell, 70% against 50%. Quality mode’s hybrid scored the same 92.5% for 1.94 cents: ten times the price for the same answers. Classic RAG did not care which mode it ran in, 60 to 62.5% across all three, against 65% on gpt-4.1. When the design is the ceiling, the model router is choosing how much to pay to hit it.

The model router cannot fix the shape. Look at the cells. Through every mode, GraphRAG scored 100% on the connected, worked-out cell and 0% on the trap, and Classic RAG scored 100% on the written-down cells and 0 to 10% on the connected, worked-out one. The same cells, with the same owners and the same orphan, as on every fixed model in two articles. The model router picks a brain. It cannot pick a store, and it cannot know that action items were never loaded into the graph. That decision is upstream of it, and it is the question router’s.

Latency needs an asterisk, and the asterisk is instructive. The median time for the agent designs through the model router came out at 27 to 35 seconds against 3 to 5 seconds on fixed models. That number is real and it is not the models’ fault. Nearly every agent step took 13 to 15 seconds, regardless of mode or served model, while plain search calls took two to four. When I disabled the client library’s silent retries and ran the same calls again, they failed instantly with a token rate limit for the served model on the router resource. The agents’ prompts, with the ontology and the tool definitions, exhaust the quota of a pool model in a few calls; the SDK waits and retries, and the wait shows up as model latency. In a clean micro-benchmark on an idle quota the router answered the full ontology prompt with tools in 1.5 to 2.5 seconds, the same as gpt-4.1. So the measurement is correct, the explanation is quota, and the lesson is operational: a model router shares each pool model’s rate limit with whatever else uses it, and you will not see that in the response, only in the clock.

Where that leaves the model router in this stack: underneath, not in front. It is a good way to stop paying frontier prices for routine calls, and in cost mode it beat a fixed mid-range model on both price and score. It is not a routing decision in the sense this article is about, because it does not know what the question needs, only how hard the prompt looks. Put the question router first. Let the model router decide how much brain the chosen design gets.

The stack, then

Put the pieces in order and the architecture is short.

1. A question router in front, small, cheap, structured output, with the shapes and the designs in its prompt, and a line for each thing your graph holds. Give it the rule to over-route when a question needs something the graph lacks.
2. Three designs behind it. Search for pages, a graph with an ontology for structure, an agent over both for the questions that need both.
3. A fallback on results, not on confidence. An empty query, a search that misses the entity, an agent that answers without a tool: escalate to the hybrid and pay for both. On this data that rule alone recovered every point the router had lost.
4. A model router underneath, if the price is right for your call mix. It decides how much brain each call gets. It does not know, and cannot be told, what the question needs.
5. Collect the questions, keep the routing log, and read the misses. Every misroute in this article was found by comparing the router’s choice with what the designs actually did. That comparison is free once both are recorded, and it is the only calibration the router will ever get.

The honest caveats

Thirty-two answered questions and eighteen routing-only ones, on one invented company. The strategies were replayed over recorded gpt-4.1 outcomes, not re-run, which is exactly why they are comparable and also why they inherit every quirk of those runs, including the judge’s strictness on one ambiguous lookup. The oracle is defined from the same outcomes and is the ceiling for these questions, not a benchmark. The 18 routing questions have expected routes that I decided from the data; the two that could be argued either way are marked in Table 2 and counted strictly. The data hint was written knowing which facts the trap questions needed, which is what the second article’s first step tells you to do, and also a reason to believe it would be less sharp written cold. And the escalation rule was replayed, not deployed: it is a count of rows, so I expect it to behave, but I have not watched it in production.

Closing remarks

The router is where the money is decided, and it is almost free. On questions the designs can answer, a hundredth-of-a-cent decision matched the best design’s accuracy at a quarter less cost, and the oracle shows there is more to take.

A router cannot route to a design that does not exist. Every point it lost was in the cell nobody owned. Give the router a list of what the graph holds and it steps around the trap; build the structure and the trap is gone, and the router beats the hybrid outright.

Do not trust the confidence. Trust the result. Every miss came with a confidence of 0.86 or higher. The design that was wrongly chosen knew it had nothing, and said so, and that is the signal to fall back on.

So, one sentence: route first, measure the routes, and when a design comes back empty-handed, let that be the dial.

Naissus Pay, its people, systems and incidents are invented for this work. Prices are Azure list prices as of 30 September 2026. Four models in the model-router pool (gpt-6-luna, gpt-6-sol, gpt-6-astra and grok-4–1-fast-reasoning) have no Microsoft price page I could find; their rates are the Global Standard figures reported by third-party trackers. The first two articles are here and here.