Decided Elsewhere
Every specification leaves something unsaid, and a running system does something in every case anyway. Between the two, something decides. A specification format is, among other things, a hypothesis about where that decision should happen and who should make it.
I proposed one such hypothesis this month.¹ Gherkin, the most common format for business-readable acceptance criteria, has no place for a guard: a condition that relates the state of a system to the parameters of the event that arrives, such as whether a cash machine can make up the requested amount from the notes it holds. The proposal, Guarded Gherkin, adds two keywords. Provided states the guard. Otherwise states what happens when it does not hold, including the frame condition that nothing else changes. Named conditions go into a glossary with a definition and an owner. The claim was modest: the format makes errors in a specification visible early, before an agent fills the gap with a guess. It also assumed a specification that is revised as findings come back. Findings came back.
They came from a test I had not planned. Afterwards, and for reasons that had nothing to do with the proposal, I built a small service with coding agents through OpenSpec, one of the spec-driven tools the proposal had been set against: about 5,200 lines of Go and some 8,000 lines of tests, built over 21 spec-driven changes. The build went well, and the service does what it should. Its record was not written in Guarded Gherkin, nor with any test of the format in mind. The two met only when I asked the format a question. Could a goal per capability, Guarded Gherkin and a glossary specify a running system, derived from its code alone?
An AI agent did the reading. It knew the proposal and the Go sources, and nothing else: not the OpenSpec record, not the design documents, not my intentions. I set the questions and the rules of method, and I questioned and decided what followed. In a second step, another agent was told how both versions had come into existence and compared the derived specification with the system’s own record. It had the sources at hand, so every contradiction it reported could be checked against the code and the archive, not merely against the other text.
What came back divides into two failures. Some decisions in the system were made by nobody. Others were made by somebody and never reached the specification. The discipline of the guard reaches the first kind. It does not touch the second.
What the other case surfaced
The reverse test produced 47 specification files in nine areas, one capability each: 256 scenarios, 97 Examples tables and a glossary of about 130 terms. Of the scenarios, 108 carry a guard. Every one of those had to say what happens otherwise, and every guard needed examples on both sides of the value that flips it.
That discipline produced 34 open questions, spread over 22 of the files. About half came directly from writing an Otherwise. A guard on a parameter, once its Otherwise has to be written, forces the question of what the system does with a value nobody expected, and the code’s answer was often a surprise. The other half came from building Examples on both sides of a guard: a value that only just satisfies it, and one that only just fails. Both are what the proposal asked of an authoring agent. Neither would have surfaced from a format that leaves the negative case and the boundary to the reader.
The questions are not all of one kind. About four are likely defects, cases where the code’s own comments describe behaviour the code does not have. One listing reports a total that counts only the entries after the paging cursor, so the total shrinks from page to page. A comment promises that an absent description can be told apart from an empty one, and the code cannot tell them apart. More of the questions concern behaviour that is consistent but surprising: an unrecognised value for a direction parameter silently means “both”, and asking a command for help ends with an error status. The rest are edge cases worth a decision. About four, as the comparison later showed, were not open at all. They had been decided on purpose, and the code gave no reason.
None of this was hidden. Each case is visible to anyone who reads the code with the right question in mind, and the format supplied the question. On the cash machine it looks like this: with one $50 note and three $20 notes, the machine can pay $90 but not $80. Provided it can make up the amount, it pays; otherwise it refuses, and nothing else changes. The Otherwise line reads like filler until someone has to write it for a guard whose failure nobody has thought about. Then it is the line that matters.
Nobody decided
The comparison answered a question the reverse test could only raise: who made these decisions? The proposal had answered it in advance. It said the unstated state is what a model fills in from its training data.
For the edge cases recorded nowhere, the answer was mostly nobody. The behaviour is the default of whatever the code hands the work to. A key of the wrong type in a document’s metadata rejects the whole document, because the YAML decoder raises a type error and the parser treats every error as unreadable input. A Windows path such as C:/notes.md counts as an external link, because the URL parser reads the drive letter as a scheme. An empty body disappears from a response, because the JSON encoder was told to drop empty fields, a convenience applied to every field alike. A file extension in capitals is accepted by one function and kept in the document's identifier by another, because two call sites made two choices.
Where an agent saw a gap, it decided, and usually wrote down why, though not in the specification. A test for links that point only to a heading within the same page carries a comment explaining that they count as dangling by design. A rule about bare directory names is argued at length in a design document. The reasons survived, in places a reader of the specification never looks. That is where the second failure begins. The silent decisions, by contrast, are the ones nobody saw. The format did not ask “and otherwise?”, so the library answered.
This corrects the proposal, though less than it seems. The gap was not filled by a judgement about the behaviour at all. It was filled by a delegation nobody examined. The agent chose the library and the data type, and training data plausibly shaped those choices. The library chose the behaviour. The decision was made at one remove, by whoever picked the tool, about a case the tool’s authors never knew would matter.
Parnas and Clements gave, as one reason for writing requirements down, keeping programmers from making requirements decisions by accident, and observed that most requirements documents leave the handling of undesired events to the programmer.² Forty years on, the undesired event has moved further down. In 1986 it fell to the person writing the code. Here it fell to a default chosen by people who never saw the system. A format that forces the other case forces someone to decide before a library does.
What the format could not hold
The verdict on the format itself is short. For behaviour, it held: nothing the system does had to be left out because the format could not express it. For the system as built, it did not, and it was never meant to. Guarded Gherkin, with its glossary, improves the semi-formal description of behaviour. That is a part of a specification, not the whole. The test measured how large the rest is.
The rest lies at two distances. The nearer one is still inside behaviour. Of the 256 scenarios, 148, or 58 per cent, carry no guard: definitions and derivations, orderings, the layout of reports and messages, timelines. Plain Gherkin carries them well. The extension adds nothing to it, and these are most of what an application is. The guard was the right gap to close. It is a smaller share of a real specification than the cash machine suggests.
The further distance lies beyond the semi-formal description of behaviour, in content that had to be bent into the format or did not fit at all. Invariants are the clearest case. “The system never writes to its input” and “the same input gives the same output” are properties of every event, not examples of one. Written as Gherkin they become “Given any state, When anything happens, Then …”, which stretches the example form past what it was made for. The cash machine has one as well: no sequence of withdrawals, refills and cancellations ever leaves it paying out a note it does not hold. No single rule states that, and every rule must respect it.
Data structures had no slot either. Which fields a response carries, and when a field is left out, turned into long chains of “And it carries …” lines that a schema would state in a few. Time and concurrency became narratives inside When steps: changes at these moments yield one reload at that moment. External standards, such as a query syntax, a markup standard or a protocol, could only be named, although anyone rebuilding the system depends on them. Rationale had nowhere to go below a sentence or two of intent per capability. The code was unusually rich in reasons, and almost none of them survived the move into the rules. The first instinct was a keyword for them, a Because line inside the rule. It was rejected. A reason is not a step, and placing it among the steps invites someone to test it as one.
Two places strained without breaking. In batch processing, such as reading every file in a folder, there are no event parameters in the usual sense, and the line between a Given and a Provided became a framing choice. It holds once the item being processed is treated as the parameter of the event. And the rule of one guard per rule turned a five-way classification into a chain of four rules, each handing over to the next in its Otherwise, linked only by names written in prose.
Build and dependency constraints never appeared at all. They are not behaviour, and the format never claimed them.
What the code could not yield
The comparison runs in both directions, and the other direction belongs to the record. OpenSpec’s specifications state their reasons inline: the 75 requirements in this system’s main specifications use “because”, “so that” or “since” 44 times, and the archived designs add whole sections of alternatives considered and rejected. The record also says what must never be added: no configurable list of paths to ignore, no query language of the system’s own, no state kept between calls. It names the intended audience and what is permanently out of scope. None of that can be recovered from code. A requirement that something must never exist looks, in the code, exactly like a feature nobody has written yet.
The cost of that blindness showed in the reverse test’s own results. About four of the 34 open questions were decisions the record had made deliberately. Broken links inside index files go unreported on purpose, because an index is not one of the documents the system checks. A bare directory name resolves to a file by design, and a design document argues why. They looked doubtful only because the code carried no reason. Reverse engineering cannot tell a deliberate decision from an accident. Rationale is what tells them apart.
Michael Nygard described the position a reader is left in. Without the reason behind a past decision, a newcomer can only accept it blindly or change it blindly.³ An agent asked to tidy up this code would have been that newcomer for those four questions, and its tidying would have been four blind changes. Parnas and Clements recorded every alternative they considered and rejected, with the reason, so that a later maintainer could find out why.⁴ This system’s record did much of that. The derived specification, for all its precision, could not, because the code it was derived from did not say.
Get Bruno Baketarić’s stories in your inbox
Join Medium for free to get updates from this writer.
So the comparison does not crown either side. The derived specification holds decisions the record never stated, and the record holds reasons and limits the code never showed. Each holds what the other lacks.
Decisions that never arrived
The second failure is the one the guard does not reach, and it needs a word on how OpenSpec keeps its record. A project has main specifications, the consolidated statement of what the system does. Each change gets a folder with a proposal, a design, a task list and delta specifications: the requirements the change adds, modifies or removes. When a change is complete it is archived. Its deltas are folded into the main specifications, and the folder, design and task list included, moves into an archive.⁵
The comparison drew up 71 decisions that the code makes and the main specifications do not state, and searched the 21 archived changes for each. One turned out to be stated in a main specification after all. Sixteen are decided in an archived design, task list or proposal: the weights that rank search results, default limits, a one-second pause before reloading, timeouts. Thirty-two are touched there without being decided. Twenty-two appear nowhere. The sixteen were chosen deliberately and reasoned well. They exist, but only inside the change that introduced them, in documents the archive step leaves behind. The external contract (every operation name, argument and response field a client depends on) is not named in the main specifications at all.
This is structural, not a lapse. A design document is where a change works out its decisions, and archiving keeps it as history. The trouble is that the history is the only place the decision lives. Three patterns sit behind the six places where the record contradicts the code or itself.
The first is change-local reasoning. An early change added a component that watches the folder for edits and triggers a reload. A later change introduced a rule that hidden directories are skipped, and argued it thoroughly for the component that reads the files. It never mentions the watcher, which still watches every directory, so activity in a hidden version-control folder triggers reloads. Each change reasoned well about itself. Nothing in the process asked which earlier mechanisms a new rule affected.
The second is that designs are not read again once archived. One design states that the specification requires an absent description to be distinguishable from an empty one. No version of the specification ever required that for descriptions, only for tags. The design invented the requirement, a code comment copied it, and the data type could not implement it: this is the comment the reverse test had flagged as a likely defect. Another design gives a reason for the order in which a link target is decoded and its trailing fragment cut off, and the reason is backwards. The order it defends is what breaks an encoded “#”, so a file whose name contains one cannot be linked at all. The main specification adopted the order. A wrong reason became the specification of record, because it sat in a document nobody read again, apart from the rule it justified.
The third is that tests became the specification for edge cases. The record says the starting point of a neighbourhood appears in it only if it is “genuinely reachable from itself”. Read literally, that holds in every neighbourhood that ignores the direction of links, since any link leads back. The implementing agent narrowed it sensibly to a document that links to itself, tests pinned the narrowing, and the specification never said so. A reader of the record cannot know the behaviour. A reader of the tests cannot know it was a decision.
Parnas and Clements stated the condition these patterns violate. No design decision counts as made until it is in the documents, and a change that invalidates a document requires that document, and every one after it, to be revised as if the change had been the original design.⁶ They asked for a record of what is decided, presented as if it had been decided rationally. OpenSpec’s archive keeps something else of value, the history of how it was decided. The gap is that the decisions stayed inside the history.
The proposal’s own pipeline has the same gap, drawn in plain sight. Findings flow back to authoring from two places, the conflict checker and operation. Nothing comes back from the coding agent or the tests. Whatever is decided while building has no way into the specification, and the conflict checker, however good its solver, can only compare what the rules say. A decision that never arrived is invisible to it.
The proposal said checking belongs at the point of change: that rule and its neighbours. The evidence shows what a neighbour is not. The watcher was not near the file reader in any document, and it was not part of the same change. It shared a notion with the new rule, which files count as part of the input, and that was enough to be affected. The condition can be stated without a mechanism: the effects of a change on existing decisions have to be searched for, not remembered. Implementation produces information that a specification written beforehand could not contain.⁷ This system shows where that information went: into code, tests and archived designs, almost never back into the specification of record.
The contradictions surfaced because the derived specification was a second reading, independent by construction: the agent that wrote it never saw the record. That is the argument the proposal made for keeping the test agent and the coding agent apart, seen from the side of the specification. Two independent readings can be compared, and a single reading cannot be checked against anything. The derived specification itself had one reader, and nobody has read it a second time.
Names before code
One gap belongs to the proposal alone, and it came out of the discussion that followed the findings, not out of the test. The proposal kept the test agent and the coding agent apart: each reads the specification, and neither sees the other’s output until both are done. But a test has to call something. Before the coding agent starts, the functions, types and modules it will create have no names, and the test agent cannot call what does not exist yet. Independent tests can only attach where an interface was named before either agent began. That makes the interface part of the specification, and the proposal never said so.
On the cash machine, a test agent can check that $80 is refused only if it knows how to ask for $80 and what a refusal looks like. Whoever fixes those two things fixes part of the contract. If nobody does, the coding agent invents them, the test agent has nothing to call, and the arrangement falls back to the single reading it was built to avoid.
What this does not show
The evidence is narrow. It comes from one small system. The derived specification had a single reader, and errors in it would go unnoticed. The test was not blind: the reading agent knew the format and was asked whether it could hold the system, not whether people writing ahead of the code would produce it. Some of what is said here about libraries rests on knowledge of how they behave, not on observing them.
Precision needs the same caution. The record’s main specifications hold 351 scenarios, roughly one concrete case each. The derived specification holds 256 scenarios and, with its Examples tables, about 750 concrete cases. Part of that difference is hindsight, since the derived version was written with the code open and the record before the code existed. What the comparison shows is narrower than one format being more precise than another: the discipline found the gaps where it was applied. Whether a business author, writing ahead of the code, would find the same ones is untested. So is the purpose the proposal gave the format, a description from which an agent derives tests that check something, because no one can read the volume of code agents produce. The test derived no tests. Each of the 34 open questions marks a case where a test agent would otherwise have had to invent the expected outcome, the oracle problem the proposal described, but that is a reading of the evidence, not a measurement.
I am working on a skill for the business side and on tooling for development, both built on Guarded Gherkin and on these findings. Neither has been tried on a real change yet, and both will have to pass the kind of test the proposal has just been through.
The two failures are not equally hard. A decision nobody made can be forced: write the Otherwise, and someone has to answer before a library does. A decision made in the wrong place cannot be forced from inside a rule. It was made, and often reasoned well, and it stayed where it was made. A format can make someone decide. It cannot make the decision arrive.
P.S.: Since writing this essay, I’ve been building tooling around an extended, yet still Guarded Gerkin based spec format. And I use the format itself to do so, which is the first proof of whether it has utility and can carry an application.
P.P.S: It’s public.
¹ See also “Provided, Otherwise: The Missing Guard in Gherkin”, which introduces the format, its two writing rules, and the pipeline of authoring, test and coding agents that this essay reads against the findings.
² PARNAS, David Lorge and CLEMENTS, Paul C., 1986. A rational design process: How and why to fake it. IEEE Transactions on Software Engineering. SE-12(2), pp. 251–257. DOI 10.1109/TSE.1986.6312940. Avoiding requirements decisions made accidentally while designing is among the reasons for a requirements document (p. 252); most requirements documents ignore undesired events and leave the decision about what to do to the programmer (p. 253).
³ NYGARD, Michael, 2011. Documenting architecture decisions. Cognitect blog. 15 November 2011. His records are scoped to architecturally significant decisions. The decisions lost here were mostly behavioural (defaults, names, edge cases), which fall outside that scope.
⁴ PARNAS and CLEMENTS, A rational design process, p. 256.
⁵ OpenSpec, Fission AI: github.com/Fission-AI/OpenSpec. The project’s README shows a change folder holding a proposal, specifications, a design and tasks, and archiving as moving the folder to openspec/changes/archive/ while the main specifications are updated.
⁶ PARNAS and CLEMENTS, A rational design process, pp. 254–256. A change that invalidates a design document requires it and all later documents to be revised as if the change had been the original design (pp. 254–255); every fact has one, and only one, place (p. 255); no design decision is considered made until it is incorporated into the documents (p. 256). The paper describes an ideal process that, by its own account, no project follows. What is borrowed here is the state of the record, not the order of work.
⁷ See also “The Spec Is Not the Method”, on why contact with implementation produces information a specification written in advance cannot hold, and what follows for spec-driven development.
