What Breaks When the World Moves and Your Ontology Does Not

The question that comes after the demo

Every time the demo ends, the same person stands up. Not the engineer; the one who will own the thing. The question is some version of: “And who maintains this?”

It is the best question in the room, and for five articles I have answered it with a sentence. The catalogue feeds the graph, the wiki feeds the pages, somebody owns the ontology. True, and useless, because it describes a company on the day of the launch. A company is not a snapshot. Services get retired, dependencies get added, a platform team renames a field, compliance narrows a rule after an audit. Every one of those changes lands in one place first and in the others later, or never. And the assistant keeps answering.

So this article does what the earlier ones did not. It lets the world move. Four kinds of drift, each one a thing that actually happens, applied to the same company, the same three designs and the same judge. The question is not which design is best. It is which one breaks, how loudly, and what it would take to notice.

Three things drift at three speeds

There are three artefacts between the company and the answer, and they are updated by different people on different schedules.

The catalogue feeds the graph. In most companies it is a system of record with a sync job, so it is usually current, until the sync breaks. The wiki feeds the pages. It is updated when somebody remembers, which for a service page is usually never. The ontology tells the agent what the graph means. It is updated when somebody notices it is wrong, and the whole point of this article is that nobody does.

Four scenarios follow from the gaps between them.

For the first two, four things happen at Naissus Pay in one quarter. The checkout API starts sending receipts, so it gains a dependency on the notification service. Onboarding starts charging a setup fee, so it gains a dependency on the payments API. A run book is registered for the 3-D Secure service. The loyalty program is retired and its service switched off. In drift 1 these land in the catalogue and the pages stay as they were. In drift 2 the pages are rewritten and the catalogue sync is broken. In drift 3 nothing happens to the facts at all: the platform team renames three things in the graph, tier to criticality, customer_facing to is_customer_facing, DEPENDS_ON to USES, and the hand-written ontology still uses the old names. In drift 4 compliance narrows the PCI scope rule after a tokenisation audit: only services that read or write a cardholder data store are in scope; the direct callers of the vault, which the old rule included, are out. The policy page is updated. The ontology's rule is not.

Each scenario got its own Neo4j database, and a search index built from its page state, loaded from the exact state the scenario describes, so the demo stayed intact and nothing was hand-edited. For every question, the ground truth was recomputed from the true state of the company, not from either store. The first two scenarios also ran in their two bracketing states, nothing updated and everything updated, so the measurements have a floor and a ceiling. All runs on gpt-4.1, 203 of them.

Drift 1 and 2: the store that moved is the design that is right

Six questions changed their answer when the four things happened: three lookups, two chains and an absence. The third lookup is the plain question of whether the wallet app still shows loyalty points.

Read the two middle groups. When the graph is current and the pages are stale, Classic RAG scores 8% on the changed questions and the graph agent 75%. When the pages are current and the graph is stale, RAG scores 50% and the graph agent 25%. Neither store is more trustworthy. Each is exactly as good as its last update, and the design on top of it inherits that, completely and silently.

Silently is the word that matters. Here is Classic RAG on stale pages, asked whether the wallet app still shows loyalty points, a quarter after the program was retired:

Yes, the wallet app still shows loyalty points. The service that provides them is loyalty-service. wallet-app-bff shows points from loyalty-service [1]. loyalty-service awards and redeems loyalty points for wallet activity [2]. Both services are owned by the Wallet team [3].

Three citations, all accurate to the page, all wrong about the company. And here is the graph agent on a stale graph, same question:

Yes, the wallet app (wallet-app-bff) still shows loyalty points. The service that provides loyalty points is loyalty-service. wallet-app-bff depends on loyalty-service for this feature.

It ran a query, the query matched an edge that no longer exists in the company, and it answered. Across the 24 wrong answers the three designs gave on changed questions in these two scenarios, exactly one carried any sign of trouble, and that one was an empty result from a mistyped name. Zero hedges. Stale data does not look stale from the inside. It looks like data.

The hybrid follows the graph. This is the finding I did not expect. The hybrid has both stores. When the pages were stale it scored 92%, because it went to the graph. When the graph was stale it scored 25%, the same as the graph agent, because it went to the graph. On the six changed questions with a stale graph, it did not search the documents once. It had the current answer a tool call away and never looked, because its instructions, and its training, say that structure is the authority for structure. The only times it reported a discrepancy were when the graph itself said so: in the two states where the graph was current, the retired service carried the word “retired” in its node summary, the hybrid read it and wrote “However, loyalty-service is marked as RETIRED”. The signal came from the store, not from the agent comparing stores.

With both stores current, the graph agent and the hybrid score 92% and Classic RAG 50%, which is the wall from the fourth article: RAG gets the lookups and loses the chains, whatever the state of the pages.

Drift 3: a rename is the loudest failure, and the easiest fix

Three names changed in the graph. No facts, no pages. The hand-written ontology still said tier, customer_facing and DEPENDS_ON.

With the stale ontology, the graph agent scored 30%: the three questions that touch none of the renamed names, and nothing else. Six of the other seven produced a query with the old names, zero rows, and a confident sentence. The seventh, which teams own no Tier 1 service, returned every team in the company, because the property it filtered on no longer existed and so no service matched. Four of the confident sentences:

No customer-facing services are affected, directly or indirectly, if card-vault-db goes down. No services are affected, directly or indirectly, if events-stream goes down. There are currently no services with a dependency on events-stream. All Tier 1 services have a run book. None are missing one. No services are currently in PCI scope according to the knowledge graph.

Four answers a compliance officer or an incident commander would act on, each the opposite of the truth, each delivered without a flicker. The hybrid did the same, 30%, and in all ten runs it never once fell back to the documents, which still said, page by page, which services depend on the event stream. An empty result from the graph was accepted as the answer.

But look at the right-hand panel. Six of the ten runs had a query that returned nothing. The failure is silent in the answer and loud in the trace. In the third article I argued that the fallback should trigger on the design’s result, not the router’s confidence. This is the scenario that argument was waiting for: an empty-result rate that jumps from zero to sixty percent overnight is a smoke alarm, and it goes off before anyone reads a wrong answer.

Two repairs were measured. Replacing the ontology with a schema introspected live from the database, which is what a team without an ontology would have, took the graph agent to 50% and the hybrid to 60%. The agent read USES and is_customer_facing from the database and used them, and that recovered the card-vault blast radius and the checkout-api lookup. It did not recover the event-stream blast radius, where the query walked the new relationship and swept in services that merely share a dependency with the stream's consumers, 21 services instead of 17. It did not recover the tier questions, because the introspected schema says a property called criticality exists and nothing about its values, so the agent guessed 'Tier 1' as a string and matched nothing. And it did not recover the definitions, because meaning was never in the database: asked which services are in PCI scope, it looked for a property that says so, found none, and answered none. The second repair was the obvious one: three find-and-replace edits in the ontology, which took the graph agent to 95% and the hybrid to 95%, with the empty-result rate back to zero. The cheapest drift to cause, the loudest to detect, the cheapest to fix, and the one most likely to happen, because the people who rename fields are not the people who own the ontology.

Drift 4: the policy changed, and the ontology kept the old one

Compliance narrowed the scope rule and updated the policy page. The ontology’s rule, with its query pattern, still encoded the old one. Nothing in the data changed, so with the stale rule no query came back empty, and no trace carried a signal.

Three things happened, and together they are the sharpest result in the article.

Classic RAG found the new policy page and, asked what the rule is, quoted it correctly. Asked which services are in scope, it applied the new rule to the five passages it had and named two services, one wrong. Asked whether the payments API is in scope, it reasoned that exchanging card details for tokens “implies it reads cardholder data” and said yes. It knew the rule and could not run it over the data, which is the wall from the fourth article, now with a policy change on top.

The graph agent, with the stale rule, returned the old five services, said yes to the payments API, and recited the old rule when asked what the rule is. Three confident answers, all superseded, all delivered from a rule that had been retired by the policy it was meant to implement.

And the hybrid, which had the new policy page one search away, did exactly what the graph agent did. On all five questions it never called the document search. Asked “what is the current rule for deciding whether a service is in PCI scope”, a question whose answer is a page, it answered from the definition in its own prompt. The ontology had become a cached copy of the policy, and the agent trusted the cache over the source even when asked for the source.

Updating the rule, one sentence and one query pattern, took both agents to 75%: the scope list correct, the teams correct, and two partials. Asked whether the payments API is in scope, both said no, correctly, but could not say why, because the reason, that the audit showed it only ever handles tokens, is on the policy page and not in the graph. Asked for the rule itself, both gave the updated rule in the ontology’s words, “depends on a cardholder data store”, which the judge held to be less precise than the policy’s “reads from or writes to”. Both partials point the same way: the ontology can carry the rule but not its justification. When the rule changes, the “why” lives on the page, and only a design that reads the page can give it.

What would have noticed

Put the four drifts side by side and ask, for each, what signal existed before anybody read a wrong answer.

Classic RAG: one hedge in 24 wrong answers, and that one in the rename scenario, which did not touch its pages at all. The graph agent: seven signals in 24, all of them empty results, six in the rename scenario. The hybrid: eight in 23, same kind. Outside the rename, the designs failed in near-complete silence: of the 42 wrong answers across the three fact-drift states, three carried an empty result, all on the loyalty question and all from the agent trying a service name that does not exist in the graph; of the nine in the policy drift, none carried anything.

So the honest answer to “who maintains this” has three parts, and they are different for each kind of drift.

Fact drift is detectable by comparison, and only by comparison. No single design can tell that its store is stale. But two stores that disagree can. On the three changed lookups, Classic RAG and the graph agent gave different verdicts in every state where one store was stale, and the same verdict in the state where nothing had been updated; with both stores current they agreed on two of the three, the exception being the checkout-api lookup where the graph’s answer includes a cache the judge does not accept as a service. That is a detector a hybrid could run for the price of one extra search: when the page and the graph disagree about a lookup, one of them has moved, and the answer should say which one it believed and why. The hybrid in this experiment never ran it. It should.

Schema drift is detectable from the trace, immediately. An empty-result rate on graph queries is a one-line metric, and in the rename scenario it went from zero to sixty percent. Alert on it. The repair is three renames in the ontology, and it should be a step in the same change that renames the field, which means the ontology has to live in the same repository as the schema, with the same review.

Definition drift is not detectable by the system at all. No empty result, no disagreement between stores that the agent would consult, no hedge. The only signal is in the policy document, which the agents did not read because they already had a rule. The fix is organizational: every business definition in the ontology carries the policy it was derived from and a date, and a change to that policy opens a ticket against the ontology. It is not clever. It is the kind of thing that is only missing because nobody wrote it down.

For the person who owns the thing, that is the maintenance contract. A sync job and a freshness check for the catalogue. An owner and a review cadence for the pages, which will drift anyway, so the hybrid must compare. A schema change process that includes the ontology, with an empty-result alert as the backstop. And a policy change process that includes it too, with no backstop, because there is none.

The honest caveats

Four drifts, on one invented company, each with five to ten questions. The changes were chosen to flip known answers, so the effects are clear by construction; a real quarter’s drift would be messier and some of it would be harmless. The hybrid’s behavior, trusting the graph for structure and the ontology for definitions, is partly a consequence of its prompt, which tells it that structure lives in the graph; a prompt that said “when a lookup is cheap, check both and report disagreement” would change the fact-drift results, and that experiment is the obvious next one. The disagreement detector in the last section was computed from the recorded verdicts, not deployed. And the introspected-schema variant is one rendering of what a database reports about itself; a richer one, with sample values, would probably have recovered the tier questions.

Closing remarks

Every design is exactly as current as its last update, and none of them knows it. Stale pages gave cited, fluent, wrong answers. A stale graph gave queried, exact, wrong answers. The hybrid, with both, followed the graph and never checked.

Drift has three kinds, and they fail differently. Facts drift silently and can only be caught by comparing stores. Schema drift fails loudly in the trace and quietly in the answer, and is fixed with three renames. Definitions drift in total silence, because the ontology becomes a cache of the policy, and the agent trusts the cache.

The maintenance contract is the architecture. A sync and a freshness check, a comparison when stores can disagree, an alert on empty results, and a rule that no policy changes without its definition. None of it is expensive. All of it is missing from every demo, including mine.

So, one sentence for the person who stood up at the end: the system will keep answering after the world moves; the only question is whether you will find out from a metric or from a customer.

Naissus Pay, its people, systems and incidents are invented for this work. The earlier articles in this series are one, two, and the three that followed on the router, the four shapes of the wall and the layers of an ontology.