Written Down or Worked Out? The Two Questions That Decide Your AI Architecture

A rule I had never measured

In the first article I argued that questions have a shape, and that the shape, not the model, decides whether a retrieval system can answer them. Near the end of the talk that article came from, there was one slide I had drawn from experience rather than from data. Two questions, four cells. I called it “my rule”, because that is what it was: the way I decide, after years of shipping these systems, what to build for a given company.

Here is the rule. Before you choose a vector database, a graph database, a model or a framework, ask two things about the questions your users will bring:

1. Is the answer written down somewhere, or does it have to be worked out? A procedure in a run book is written down. “Which of our systems are in audit scope” is worked out: a rule applied to everything you own.
2. Where does the knowledge live: in text, or in connected data? A wiki page is text. A service catalogue, an org chart, a ticket tracker are connected data, whether or not anyone has ever called them a graph.

Two questions, two answers each, four cells. Each cell has a design that owns it, and one of them has a trap in it. I have drawn this on whiteboards for clients more times than I can count. What I had never done was test it. This article is the test.

The four cells

Laid out as a matrix, the rule looks like this.

Text, written down. Someone wrote the answer, and your job is to find it. This is Classic RAG, and it is the cheapest thing that works. If every question your users ask lands here, stop reading and ship it.

Connected, written down. The answer is a single fact: who is on call, which team owns this, what this service depends on. It lives in structured data and it is also, usually, repeated on some page. Any design answers it. The point is that none of them should be paid for it. This is a lookup, a form field, an API call. The language model is at most there to write the query.

Connected, worked out. What breaks if this database fails. Which teams own nothing critical. Which services touch restricted data. The answer is not in any record; it is between them, or it is a rule applied to all of them. A graph with an ontology owns this cell, because a database can traverse, count and compare exhaustively, and a language model cannot.

Text, worked out. The trap. The answer must be assembled from many pages, and nothing structured holds it. “How many promises from our postmortems are still open?” “Which run books cover a rollback?” “List every project we did in retail.” This is the cell my first chatbot died in, years ago, and the rule says something uncomfortable about it: no retrieval trick fixes it. Build the structure first.

That last claim is the one I most wanted to check, because it is the expensive one. If I am wrong, a bigger context window or a better reranker solves the problem for a fraction of the cost. If I am right, a lot of teams are tuning the wrong thing.

The experiment

I used the same invented company as before. Naissus Pay is a fictional payments company with 29 services, 19 data stores, 8 teams, 20 incidents with postmortems, 10 run books, 6 architecture decisions and 4 policies. Its knowledge exists twice, generated from one source of truth: as 105 wiki-style pages, chunked into 108 passages of about 17,000 tokens in total, in Azure AI Search; and as a knowledge graph of 165 nodes and 386 relationships with an ontology on top, in Neo4j. The graph was loaded from the structured source, the way a company would export it from its service catalogue. Nothing was extracted from text by a model.

Then I wrote twenty new questions, five per cell, each with a ground truth computed from the data. I placed them deliberately: not by how they sound, but by where the answer actually is. “Which postmortems have an action item about adding an alert” sounds like a search. It is not. Action items live only inside the twenty postmortem pages, and the graph, loaded from the catalogue, has no idea they exist. That is the trap cell by construction.

The three designs are the ones from the first article, unchanged. Classic RAG: hybrid search with a semantic reranker, five passages, one model call. GraphRAG: an agent that reads the ontology, resolves names and writes read-only Cypher, with no access to prose at all. The Agentic Hybrid: the same agent with four tools over both stores. All three ran on gpt-4.1, and an LLM judge with the ground truth and a list of key facts graded every answer correct, partial or incorrect. The judge is strict about sets: “exactly these five” with four listed is partial, not correct.

Two things are new. To test the claim that no retrieval trick fixes the trap, I ran Classic RAG again with 10, 20 and 50 passages instead of 5, and then once more with no retrieval at all: all 108 passages in the prompt, the brute-force ceiling. And after the first round, I did what the matrix tells you to do with the trap cell, and measured again. That comes last.

What happened, cell by cell

Text, written down: RAG 100%, hybrid 100%, graph 10%. Exactly as drawn. Freeze periods, the reasoning behind the event stream, what to do when the bank rail rejects a batch: Classic RAG found the page every time, in about two seconds, for less than half a cent. The graph agent, which holds no prose, said so, honestly and unhelpfully: “The content of the change management policy, including details about freeze periods, is not present in the knowledge graph.” It picked up a single partial on the incident question by returning the node’s metadata. The one thing I would add to the slide is a warning: in this cell, the graph is not merely useless, it is expensive and useless, at more than twice RAG’s cost per question, because the agent kept trying.

Connected, written down: everyone passes. RAG 100%, graph 90%, hybrid 90%. The one miss is instructive. Asked “which services does checkout-api call directly”, both agents returned the four services and also session-cache, which is a data store checkout-api depends on. The query was right; the question was ambiguous; the judge was strict. In the real world that is a conversation, not a defect. What matters in this cell is the bill: 0.31 cents for RAG, 0.42 for the graph agent, 0.60 for the hybrid, for a fact that a form field could return for nothing. The slide says “just a query”, and the numbers say the same: this is the cell where the chatbot should not be in the loop at all.

Connected, worked out: graph 100%, hybrid 100%, RAG 20%. The mirror image of the first cell. “If events-stream goes down, which services are affected, directly or indirectly?” The true answer is seventeen services, because the stream feeds notifications, fraud scoring, loyalty, audit logging and analytics, and those feed checkout, onboarding, chargebacks and the wallet. Classic RAG, reading the five best pages, named two and hedged a third. The graph agent wrote one query with a variable-length path and listed all seventeen. “Which teams own no Tier 1 service?” RAG answered that “the passages do not provide information about any teams that own no Tier 1 service”, which is true of the passages and false of the company. The graph agent wrote a NOT EXISTS pattern and named the two teams. Nothing in this cell can be found; it can only be computed.

Text, worked out: RAG 40%, graph 0%, hybrid 50%. The trap, measured. The graph agent scored zero, exactly as the matrix predicts, because it had nothing to query. Classic RAG, with five passages, counted four open action items where there are eleven, missed one of the five postmortems that promised an alert, and declared that no data store has a recovery time objective above two hours, because none of the three that do were in its passages.

The hybrid’s failure is the one I keep thinking about. Asked how many action items are still open, it did the right thing: it recognised a counting question and went to the graph. It wrote a query against an ActionItem label with an OWNED_BY relationship to teams. Neither existed. The query returned nothing, and the agent answered: "There are no open postmortem action items currently." The model had the right design in its head. The data had not been given the shape the model assumed. I cannot think of a cleaner demonstration of what the trap cell is: not a failure of retrieval, not a failure of reasoning, but a shape that nobody built.

Can you brute-force the trap?

The obvious objection to “build the structure first” is that context windows are now enormous. Why not just retrieve more, or retrieve everything? I tried both.

More passages help the trap cell, up to a point. Going from 5 to 50 passages took the text-worked-out score from 40% to 80%, and the cost from 0.37 to 1.40 cents per question, roughly four times. At 50 passages the model has almost half the company in front of it, and it can do a fair amount of counting. It still got the open action items wrong, in the other direction this time: thirteen instead of eleven, with two teams over-counted. Reading forty pages and tallying across them is not what language models are reliable at, and the judge noticed.

The connected cell barely moved. Twenty, thirty, forty percent. The chain question stayed wrong at every setting, because no number of pages contains the seventeen-service answer; each page describes one service and what it calls, and the model would have to hold the whole dependency graph in its head and walk it backwards. That is a traversal, and traversal is what databases are for.

And something slipped. At 50 passages the lookups, which had been perfect, dropped to 80%. Asked which architecture decisions concern the card gateway, the model now had more candidates to be wrong about, and it added a decision that merely mentions the gateway in passing. More context, more confident noise.

Then I removed retrieval altogether and put all 108 passages, about 17,000 tokens, into one prompt. Two surprises. First, it was cheaper than fifty passages, 0.99 cents against 1.40, because the corpus is identical on every call and Azure’s prompt caching billed almost all of it at the cached rate. Second, it was worse. The trap cell fell back to 60%, the lookups stayed at 80%, and on the gateway question the model produced a decision that does not exist, attaching the title of one record to the identifier of another. It also took 5.6 seconds at the median, more than twice Classic RAG. The whole company in the prompt is a worse reader than five good passages on simple questions, and an unreliable accountant on the hard ones.

So the honest answer to the objection is this. Brute force bought the trap cell forty points, at four times the price, with a ceiling of 80%, on a corpus small enough to fit in a single prompt. Your corpus does not fit in a prompt. And even here, it never reached the number a database returns for free.

Then I did what the matrix says

The rule for the trap cell is “build the structure first”. That is easy to write and expensive to ignore, so I did it, and measured.

Four things that had existed only in the pages got a place in the graph. Action items became nodes, each linked to its incident and assigned to its team, with a status. Run book procedures became nodes under their run book. Data stores got their recovery objectives and their retention, in years, as properties. Services got their availability target and latency budget. In the demo this is a loader reading the source of truth; in a real company it is four exports: the incident tracker, the run book index, the disaster-recovery register and the SLO catalogue. Every one of those systems already exists in every company that has postmortems and run books. I did not extract anything from text with a model, because I did not have to, and neither would you.

The cost of the structure: 62 new nodes, 101 new relationships, properties on 48 existing nodes, and two classes, three relationship types and one business definition added to the ontology. The ontology grew from 1,854 to 2,205 tokens. Then I ran the same twenty questions, on the same model, with the two agent designs.

GraphRAG on the trap cell: 0% before, 100% after. The same question that produced “the knowledge graph does not contain information about postmortem action items” now produced the right eleven, with the right count per team, from one query:

MATCH (i:Incident)-[:HAS_ACTION_ITEM]->(a:ActionItem {status: 'open'})-[:ASSIGNED_TO]->(t:Team)
RETURN t.name AS team, count(a) AS open_items

The retention question became WHERE d.retention_years > 5. The recovery-time question became WHERE d.rto_minutes > 120. The rollback question became a match on a procedure name. The alert question, which is a classification over free text, became WHERE toLower(a.name) CONTAINS 'alert', which is a cheap trick, and it worked, because the text was now short, labelled and attached to the thing it describes. Each answer was exact, complete, and cost about half a cent.

The hybrid: 50% before, 90% after. Its one remaining miss is worth a sentence. On the alert question it was offered both stores and chose the documents, because the question sounds like a search, and it missed one postmortem. The pure graph agent had no such choice and got it right. An agent with more options is an agent with more ways to take the familiar road.

Nothing else moved. The three cells that already had an owner scored the same before and after, which is the result you want from adding structure: it fixes the thing it was built for and breaks nothing.

The same picture on a smaller model

To check that the cells are a property of the designs and not of gpt-4.1, I ran the three designs again on gpt-5-mini, on the original graph, without the structure.

The shape is identical. Classic RAG is perfect on both written-down cells and collapses on both worked-out ones. The graph agent owns the connected, worked-out cell and scores zero in the trap, for the same reason: there is nothing to query. The hybrid did a little better in the trap on this model, 70% against 50%, because it leaned harder on reading whole documents, and a little worse on the connected cell, where it twice added an item the judge did not accept. The prices are different, about a fifth of gpt-4.1’s, and the latency is higher. The cells do not move. Which is the point: the matrix describes the data and the questions, not the model, and swapping the model changes the bill, not the shape.

The matrix, measured

Here is the slide again, now with numbers on it.

Three things to take from the cost table. Classic RAG costs the same everywhere, about four tenths of a cent, whether it is right or wrong, which is exactly the problem: it has no way of knowing it is in the wrong cell. The graph agent is cheapest in the trap cell before the structure exists, because it gives up fast; honesty is cheap. And the hybrid charges a premium in every cell, which is the price of not having decided. That premium is worth paying for questions that straddle cells, and worth routing around for everything else.

What this means if you are deciding

I write these for two readers: the engineer who will build it, and the person who will pay for it. For both, the matrix turns into a procedure.

Start by sorting the questions, not the data. Take a hundred real questions from your tickets, your chat logs, your on-call channel. Put each one in a cell. In my twenty, five were pages. In a real log the proportion is not kinder, and the ones that hurt are the ones that sound simple: “how many”, “which ones”, “list all”, “is anything missing”. Those words mean “worked out”, and worked out means a database, not a search.

The written-down cells are nearly free. Treat them that way. Classic RAG owns one of them, and a plain lookup owns the other. If a vendor is charging you agent prices for “who owns this service”, you are paying for a decision that was never needed.

The connected, worked-out cell is where a graph earns its keep. Not because graphs are fashionable, but because the question is a traversal, a count or a rule, and those are database operations. The ontology is what lets a model turn the question into the query; without it, as the first article showed, the same graph is blind.

The trap cell is not a retrieval problem, and it will not respond to retrieval budgets. More passages took it to 80% at four times the cost and then stalled. The whole corpus in the prompt did worse. Structure took it to 100% for half a cent a question. The structure was not expensive, because it was not extracted; it was exported from systems you already run. The real cost is organizational: somebody has to own the fact that action items live in the tracker and the tracker feeds the graph. That person is rarely in the AI team.

When the text really is only text, when nobody has ever classified your projects by industry and there is no system to export from, then yes, you classify once, with a model if you like, as a batch job with a human checking the output. But you store the result as structure and query it forever. That is still building the structure first. It is not retrieval.

And route. Every real company is a mix of all four cells. The mature architecture is not a bet on one design; it is three designs behind something cheap that reads the shape of each question before anything runs. I described the router in the first article, and nothing here changes it, except that the trap cell now has a destination.

The honest caveats

Twenty questions, chosen to exhibit the cells, on one invented company. The per-cell results are the point; the overall averages are not a benchmark. The judge is a language model with a reference answer, and I read every verdict; the three partials that came from ambiguous questions are noted above. The whole-corpus baseline benefited from prompt caching in a way a real workload with varied prefixes would not. And the structure I added was chosen knowing the questions, which is exactly what step one tells you to do, and also exactly why you should collect the questions before you build.

One more. The corpus here is 17,000 tokens. The brute-force experiment is only possible because of that, and I ran it so that nobody has to wonder. On a corpus of 17 million tokens, there is no brute force; there are only the four cells.

Closing remarks

Two questions before any technology. Is the answer written down, or must it be worked out? Does the knowledge live in text, or in connected data? Everything downstream, the store, the design, the model, the bill, follows from those two answers. The rule I had been drawing from experience held in three cells out of four, and the fourth held once I did what the rule said.

Written down is cheap. Worked out is a database. Search finds what somebody wrote. Counting, chaining, comparing and applying a rule are not search, and no amount of context turns them into search. The measured ceiling for brute force was 80%, at four times the price. The database returned 100% for half a cent.

The trap is a shape nobody built. The agent that answered “there are no open action items” had already written the right query. It was querying a structure that did not exist yet. The fix was four exports and a loader. The hard part was knowing which four.

So, if there is room for only one sentence: before you tune retrieval, ask whether the answer was ever written down. If it was not, retrieval was never the job.

Naissus Pay, its people, systems and incidents are invented for this work. Prices are Azure OpenAI list prices as of 30 September 2026. The first article, with the three designs, the ontology and the router, is here.