Knowledge That Remembers Why — A Knowledge Governance Guide
Define Truth Once, Share Everywhere — with a Rulebook
Article 5: A Knowledge Governance Guide to Institutional Memory
Audience: CDOs, data stewards, knowledge managers, enterprise architects, semantic layer owners
This is part 5 of a 10-part series exploring what it looks like to build trustworthy AI systems with verifiable outputs.
The Organization That Can Remember Why It Believes Things
Knowledge governance is not just storing definitions. It is governing how definitions are born, tested, revised, and reused.
That distinction sounds abstract until you watch a concept fail in the field. The failure mode is almost always the same: a term that everyone uses and nobody defined the same way. The governance gap is not in the repository — it is in the lifecycle. Organizations know how to store things. They rarely know how to govern the conditions under which a stored thing can be trusted.
The project behind this article is a concrete example of that lifecycle: an Effortless Rulebook model of Simpson’s Paradox. The rulebook is the source of truth. From it, the project generates database views, documentation, an ontology, a RuleSpeak explanation layer, an explorer UI, and invariant checks. The point is not that Simpson’s Paradox is the only domain where this matters. The point is that it is a small enough domain to see the whole governance problem clearly.
Simpson’s Paradox is what happens when an aggregate result points one way, but the same data points the other way once you break it into the right subgroups. A treatment can look better overall while being worse in every subgroup. A school can look biased in the aggregate while its departments tell a more complicated story. The paradox is not magic. It is what happens when the structure underneath the numbers matters more than the headline average.
That makes it a useful test case for knowledge governance. It forces the organization to ask: what exactly do we mean by “reversal”? What exactly counts as a subgroup? What formula decides when allocation has distorted the pooled result? Which claims are true by definition, and which are merely patterns in the current corpus?
The concept drift problem every organization has
An organization defines “at-risk customer” in 2019. By 2023, three teams use the term differently. The customer success team means any customer who opened a support ticket in the last 30 days. The sales team means any customer with renewal probability below 60%. The analytics team means any customer whose usage dropped more than 20% quarter-over-quarter. Each team can defend its definition. No team is using the same definition. Every cross-functional report that includes “at-risk customer” is quietly wrong.
This is not an AI problem. But AI makes it dramatically worse. When a language model proposes a concept name, the prose sounds authoritative. It gets copy-pasted into documentation, presentations, and dashboards before anyone has asked what it actually computes. The definition exists — in a dozen places, in slightly different forms, none of them governed.
The same thing can happen in a statistical domain. A model proposes a useful-sounding predicate: IsSignFlip, AllocationDistortion, SignalPurity, CorrectedWinner. In ontology and data science terms, this is predicate invention: creating new derived concepts that let the system ask questions it could not ask before. Predicate invention is powerful. It is also dangerous if the predicate is only a name. A name is not knowledge. A name becomes knowledge only when it has a formula, a test, a dependency path, and a revision history.
The symptom that surfaces years later is familiar to every data steward: a critical metric that no two systems compute the same way, an audit that cannot find a canonical formula, a new team member who spends three weeks figuring out which definition to use.
The vocabulary lifecycle: birth, test, revision, archive
A governed concept has a lifecycle with four phases, each with explicit conditions that must be satisfied before the concept advances.
Birth: the concept is proposed with a name, a formula, and the domain question it answers. It goes into the governed artifact — the rulebook — as a typed field definition with a precise formula. Not a description. A formula. If it cannot be written as a computable expression, it is not a governed concept — it is a label. Labels drift.
Test: invariant checks verify that consequences of the definition hold across the full dataset. “Invariant” sounds more abstract than it is. It simply means: if this definition is true, then certain other things must always be true. If a study is classified as a sign-flip reversal, the corrected winner and the pooled winner should disagree. If CorrectedGap is supposed to be allocation-invariant, it should not move when the allocation weights are swept across possible case distributions. A concept that cannot survive those checks has not earned its place in the model.
Revision: when a corpus expansion breaks a prior finding, the Conclusions table records the revision. The old finding is tagged corpus-pattern-superseded with a link to the build iteration where it changed. Nothing is deleted. The organization can see what it used to believe.
Reuse: once a concept is in the rulebook, it becomes available for downstream formulas to reference. SignalPurity depends on CorrectedGap and AllocationDistortion. PolicyImplication depends on DistortionType. The dependency graph is explicit and machine-verifiable. A change to a foundational concept propagates automatically to every formula that references it — and the invariant suite confirms the propagation is consistent.
This is the difference between “we have a glossary” and “we have governed knowledge.” A glossary can tell you what a word is supposed to mean. A rulebook can show you how the meaning is computed, where it came from, which downstream concepts depend on it, and what broke the last time it changed.
The governed vocabulary of Simpson’s Paradox
In this domain, every concept in the vocabulary passed through this lifecycle. The result is a layered concept hierarchy.
Core domain objects are Study, Treatment, Stratum, and CaseCell. These are the raw inputs: the published study metadata, the treatment arms being compared, the subgroups within each study, and the raw counts of successes and cases. A CaseCell is the leaf node: one treatment, in one stratum, with a number of cases and successes.
First-order derived concepts are StratumSuccessRateA, StratumSuccessRateB, and StratumGap. These are computed directly from the case counts and form the foundation of every higher-level concept. In math, a stratum is just a subgroup. A stratum gap is the difference between the two treatments inside that subgroup.
Allocation concepts are TreatmentExposureFraction, StratumFraction, and WeightedStratumGapSum. This is the mechanism vocabulary: the concepts needed to describe how allocation distorts pooled results. In Simpson’s Paradox, the problem is often not that the subgroup math is hard. The problem is that the groups are not distributed evenly across treatments. One treatment may be overrepresented in easier cases, harder cases, larger departments, smaller departments, younger patients, older patients, or any other stratum that changes the outcome.
Classification concepts are IsSignFlip, IsStratumUnanimous, and DistortionType A/B/C+/C−/D. This is the taxonomy that categorizes what kind of distortion each study exhibits. A Type A study is the canonical Simpson’s Paradox case: every stratum points one way, but the pooled result points the other way. Type B still flips, but the strata are mixed. Type C means the pooled result does not flip, but allocation still exaggerates or compresses the effect. Type D means the pooled result is comparatively trustworthy at the observed allocation.
Epistemic concepts are SignalPurity, AllocationDistortion, and DistortionRatio. This is the measurement vocabulary that quantifies how much of the pooled result is signal and how much is allocation noise. SignalPurity is intentionally plain in spirit: real signal divided by real signal plus distortion. When it drops below 0.5, the allocation noise is doing more work than the underlying stratum-level signal.
Policy concepts are PolicyImplication, CorrectedWinner, and AdjustmentAppropriate. These are the actionable outputs that tell a practitioner what to do with a given study. The system is not merely naming a paradox. It is saying whether to stratify immediately, investigate a confounder, check allocation bias, or trust the pooled analysis.
Get eejai42’s stories in your inbox
Join Medium for free to get updates from this writer.
Every concept in this hierarchy is defined once, in one place, with a formula, a description, and a dependency path. There are no team-specific variants. No parallel definitions that diverged when the analytics team moved to a new tool. One concept, one formula, one test. That is what “governed” means at the semantic layer.
The loop history as institutional memory
The loop table — 85 completed build iterations, and growing — is the complete intellectual genealogy of how this vocabulary was built. Each row records four things: the question the organization could not ask before this loop, the concept introduced to let it ask that question, what was actually witnessed when the build ran, and the git commit that anchors the record to a reproducible state.
Three excerpts show why this matters.
Loop 6: “A: 263/350=75% in large-stone stratum. B: 80/350=23% in large-stone stratum. The 52-point gap IS the mechanism — now a first-class derived number (TreatmentExposureFraction).” This is the loop where allocation became a first-class concept rather than background context. A new team member reading this row understands not just that TreatmentExposureFraction exists, but why it had to exist — because the 52-point allocation gap in the kidney stone study is the entire story.
Loop 17: “kidney-1986: WSGSUM=+0.0537, SignedPooledGap=−0.0457 → IsSignFlip=TRUE. berkeley-1973: IsSignFlip=TRUE. Per-stratum breakdown shows type-B partial reversal.” This is when the core phenomenon was formalized. The loop record is both a design note and a test fixture: here are the two studies that were used to verify the definition, with exact numbers.
Loop 44: “All type-A/B studies have SignalPurity < 0.5 — the confound is doing more than half the work. AvgSignalPurityReversal ≈ 0.35, AvgSignalPurityNonReversal ≈ 0.70.” This is when SignalPurity became more than a proposed metric. It became a checked consequence of the model. The average reversal study has only about 35% real signal.
Later loops extended the same idea beyond individual studies. Loops 80–85 added a confounder identity layer: 19 canonical mechanism archetypes mapped across the corpus. That changed the question space. Before that layer, the model could ask whether a particular study had a reversal. After that layer, it could ask which kinds of confounders produce manifest flips, which produce latent-only risk, and which domain labels are actually different names for the same underlying mechanism.
A new team member reading the loop history from row 1 to row 85 acquires the full context: every concept, when it was introduced, what evidence justified it, and what question it was designed to answer. This is institutional memory in a form that survives personnel turnover. The knowledge is in the table, not in the heads of the people who were in the room.
The three-tier epistemic system
Not all knowledge has the same status, and a governed semantic layer should know the difference. The Conclusions table uses three epistemic tiers.
Theorem: an algebraic consequence of the definitions. True by construction. Cannot be falsified by more data — only by changing a definition. Example: “CorrectedGap is allocation-invariant.” This follows from the formula. CorrectedGap is defined from the weighted average of per-stratum gaps, using a stable stratum weighting scheme rather than the observed treatment allocation. Once the formula is fixed, changing how cases are distributed across treatments should not change CorrectedGap. The invariance is built into the definition. The invariant check confirms that the implementation obeys the definition.
Discovery finding: a pattern found in the corpus after testing a pre-registered hypothesis. Example: certain confounder-labeled studies tend to produce manifest sign-flips, while collider and selection mechanisms often produce latent-only geometry. This is a real observation in the current 238-study corpus. It is tested and recorded. But it is contingent on this particular collection of studies. More studies could change the rates. A different corpus might show different patterns.
Corpus-pattern-superseded: a finding that was valid at a smaller sample size and required revision when the corpus expanded. Earlier domain-level claims and Type-D dominance patterns changed when the corpus grew to 238 studies. These findings are not deleted. They are tagged with the loop number where the revision occurred and preserved as part of the epistemic record.
Most governance systems have no analog to these tiers. Every claim lives in the same undifferentiated repository: things we believe. When someone asks “how confident are we in this?”, there is no structural answer. The three-tier system provides one. Theorems are as confident as the definitions themselves. Discovery findings are confident contingent on this corpus. Superseded findings represent beliefs that have been revised and should not be acted on without awareness of the revision.
Decision-making quality depends on knowing which tier a given piece of knowledge belongs to.
The knowledge dashboard
The ModelSummary table provides a single-row self-portrait of the corpus. At N=238, the project records 238 studies, including 233 real studies and 5 synthetic structural controls. It records 86 sign-flip reversals, or about 36% of the corpus. The five distortion types break down as 78 Type A, 8 Type B, 8 Type C+, 7 Type C−, and 137 Type D. The model also records 2,380 allocation sweep rows, 24 active discovery hypotheses, 19 canonical confounder identities, and 5 witnessed theorem-level conclusions, with one additional theorem candidate.
This is a knowledge dashboard. It summarizes the state of what the governed semantic layer knows, at the current build, across the full corpus. It is not a report that someone wrote. It is computed deterministically from the same rulebook and corpus that drive everything else. Every time a new study is added to the corpus, the dashboard updates automatically. Every time a new theorem is promoted, the count increments. The dashboard cannot be out of date in the normal documentation sense, because it is generated by the same build that generates the database, the documentation, and the invariant check results.
This is what “governed” looks like at the knowledge layer: a dashboard that the organization did not hand-maintain and cannot quietly manipulate, because it is a derived fact from the source of truth.
The IngestionProtocol as semantic governance at the acquisition layer
A governed semantic layer is only as good as what it admits. The project’s ingestion protocol defines the conditions a study must meet before its data enters the corpus.
Required inputs include a StudyId, a StratumId per CaseCell, a TreatmentLabel, Cases, and Successes. Structural constraints include binary treatment, at least two strata, paired A/B cells, and enough cases per cell to avoid meaningless arithmetic. Inclusion criteria require the outcome direction to be declared before classification, the stratum variable to be named before the study is typed, and no post-hoc splitting to manufacture a paradox after the fact. Exclusion criteria include missing stratification variables and observational studies that make no causal or policy-relevant claim.
A study that fails any of these conditions is excluded. The exclusion is documented. The system knows why a study is absent — it is not just absent.
This is the acquisition layer of semantic governance: the conditions under which raw data is admitted to the governed layer. Most organizations have no such layer. Data arrives, gets processed, and the downstream model reflects whatever came in. When a finding looks strange, there is no systematic way to ask whether the underlying data met the conditions the analysis assumes.
The IngestionProtocol is not a filter applied to existing data. It is a contract that defines what the concept of “a study in this corpus” means. Violating the contract would change the meaning of every concept that depends on the corpus — and the invariant suite would catch the downstream inconsistency.
What the organization gains
An organization that governs its vocabulary through this lifecycle gains something rare: the ability to know why it believes things.
Not just what it believes — every organization has claims it acts on. But why: the formula that defines the concept, the test that verified the algebraic consequence, the loop number where the concept was introduced, the witness note that justified it, the tier that distinguishes a theorem from a contingent finding, and the archive that records what was believed before and when it changed.
For ontologists, the important move is that invented predicates do not float as informal vocabulary. They are grounded as computable fields with dependency paths and conformance checks. For data scientists, the important move is that feature engineering becomes auditable: proposed features are defined, tested, promoted, revised, or retired. For governance leaders, the important move is that institutional memory no longer depends on remembering who made the original decision.
Knowledge governance is not just storing definitions. It is governing how definitions are born, tested, revised, and reused. Organizations that do this well do not just have more reliable analytics. They have a foundation for durable institutional learning — the kind that survives the departure of the person who built the original model.
Also in this series:
Part 2: A Boardroom Guide to Governed AI Knowledge — the ROI of governed knowledge assets and the three governance risks
Part 6: A Legal and Compliance Guide to Auditable AI — the provenance, traceability, and documentation requirements for regulated AI outputs
Full series overview: Define Truth Once, Share Everywhere — with a Rulebook
