The Inspector’s Virtual Array
Fiene’s Differential Monitoring as Predictive Signal Reconstruction
WodansEye Research Series — Don Gunter, with Claude (Anthropic) July 2026 — Companion harnesses: fiene_tdil_check.py, fiene_etree_accel.py, e2b_scale.py
0. Audience
Three readerships, deliberately braided:
Regulatory scientists and licensing administrators — Fiene’s own audience. You already know risk assessment, key indicators, and differential monitoring. What you may not know is that your field independently discovered a general predictive-inference architecture, and that naming it buys you forty years of tooling from signal processing and machine learning.
Signal and systems engineers — TDIL readers. You know predictive channel synthesis, anomaly deltas, and confidence-weighted reconstruction. What you may not know is that a research psychologist spent five decades applying your architecture to childcare inspections, with human inspectors as sensors, and validated it against real-world outcomes at state scale.
Diamond Model readers — the social-dynamics audience. Fiene’s central empirical finding — that substantial compliance produces better outcomes than full compliance — is not a bureaucratic curiosity. It is the wisdom-ordering policy stated in regulatory language: the least destructive sufficient word. If you have ever wondered what “stability without strangulation” looks like with fifty years of licensing data behind it, this is it.
Epistemic tags used throughout: [REF] published prior work, [COMPUTED] verified by the companion harness this session, [CONJ] structural conjecture, [ANALOGY] a mapping asserted as illuminating but not proven isomorphic, [PENDING] pre-registered prediction with falsifier.
1. Fiene in One Page [REF]
Richard Fiene (research psychologist, Penn State, ret.; Research Institute for Key Indicators) is the central figure in licensing measurement — the science of how a regulator with finite inspectors should monitor thousands of licensed facilities. Four pillars:
Risk assessment. Each rule gets a risk quotient: how severe is the hazard the rule prevents, and how likely is it to occur. Severity × probability, usually elicited through expert weighting protocols and arranged in a risk assessment matrix.
Key indicators. A small subset of rules that statistically predict overall compliance. Selected by phi coefficient (the correlation for dichotomous data) between a single rule’s pass/fail and the facility’s overall substantial-compliance status. Inspect six rules, know fifty.
Differential monitoring. The synthesis: not all facilities get the same inspection. Abbreviated inspections (key indicators + high-risk rules) by default; a full inspection triggers only when the abbreviated pass raises a flag. Adopted in various forms across U.S. states and internationally.
Theory of Regulatory Compliance. The deep result: compliance-versus-outcome curves show a ceiling effect. Past substantial compliance (~roughly the 90s in percentage terms), additional enforcement buys nothing — and facilities at 100% compliance often show worse program quality than facilities at substantial compliance. Diminishing returns is not a metaphor here; it is the measured shape of the curve.
Fiene also names the field’s foundational measurement problem: licensing data is nominal. A rule is in or out of compliance — 1 or 0, no Likert middle ground — and by default every rule carries equal weight. Risk weighting exists precisely because that default is wrong.
2. The TDIL Mapping [ANALOGY → COMPUTED]
TDIL — Temporal Dendritic Inference Layer — is a signal-layer architecture: sparse, noisy, bandlimited real channels feed a predictor that synthesizes the dense virtual array an ideal sensor would have seen. Every channel carries three values: effective (cleaned observation), predicted (the virtual synthesis), and delta (their disagreement — the anomaly score).
Now read a licensing system through that lens:
Regulatory object TDIL object One rule, one facility One channel Full inspection (all 50 rules) Dense real array — expensive, rarely affordable Abbreviated inspection (key indicators) Sparse real channels Key indicator selection (phi coefficient) High-mutual-information channel selection Predicted overall compliance Virtual channel synthesis Flag raised on abbreviated visit Delta exceeds threshold Triggered full inspection Controller commits the expensive sensor Inspector The sensor itself — noisy, bandlimited, human
The claim is not that Fiene “sort of resembles” TDIL. The claim is that differential monitoring is a TDIL controller: measure the informative channels, synthesize the rest, act on the delta. Fiene derived it from phi coefficients on dichotomized licensing data in the 1970s–80s, decades before predictive coding became fashionable, because the resource constraint forced the architecture. Convergent evolution under the same selection pressure.
Computed demonstration [COMPUTED]. The harness simulates 2,000 facilities × 50 nominal rules, latent quality driving compliance, risk weights on rules, and a substantial-compliance cut at 90%. Selecting K=6 key indicators by phi coefficient — Fiene’s actual method — then synthesizing all 50 channels from those 6:
- Per-bit reconstruction of the full compliance vector: 0.904 accuracy from 12% of the channels.
- Substantial-compliance classification via virtual channels alone: 0.899.
Six real channels reconstruct the array. That is the key indicator claim, restated as channel synthesis.
The budget experiment [COMPUTED]. Fix the inspection budget at 20% full inspections. Compare delta-triggered targeting (predicted shortfall from the virtual channels) against uniform random assignment, scoring against ground-truth violators (any high-risk rule out of compliance):
- Precision: 1.000 vs 0.673. Every delta-triggered full inspection landed on a true violator.
- Recall: 0.293 vs 0.197 — half again as many violators caught on the same budget.
This is Fiene’s efficiency argument for differential monitoring, reproduced from the TDIL side in forty lines of NumPy. Same architecture, same payoff.
3. The Odd Part of the Compliance Word [ANALOGY → COMPUTED]
Take a facility’s compliance record as a binary word — fifty bits, one per rule. Fiene’s complaint that all rules are weighted equally is, in this reading, the complaint that all bit positions are treated as equal. They are not, and arithmetic already knows how to say so.
The odd-part operation strips trailing zeros from a binary integer: everything divisible-by-two falls away, and what remains is the irreducible odd core that determines the number’s multiplicative identity. The stripped zeros are structure without information — they tell you about scale, not content.
The regulatory analogue: rules that (nearly) everyone passes or (nearly) everyone fails carry near-zero entropy. They are the trailing zeros of the compliance corpus — real bits, honestly recorded, discriminating nothing. Strip them, and what survives is the odd part of the rule set: the entropy-bearing core where facilities actually differ.
Computed [COMPUTED]. Applying an entropy strip (H > 0.30 bits/rule) to the simulated corpus:
- 32 of 50 rules survive into the odd part.
- All six key indicators land inside it.
That containment is the point. Fiene’s phi-coefficient selection is a sharper filter operating strictly within the odd part — phi rewards not just entropy but alignment with the substantial-compliance boundary. The odd-part strip is the cheap first pass (per-rule, no cross-tabulation needed); key indicator extraction is the refinement. Fast odd part, then phi. A two-stage sieve, and the first stage costs almost nothing.
Get Don Gunter’s stories in your inbox
Join Medium for free to get updates from this writer.
[CONJ] The residue that has entropy but low phi is also meaningful: those are rules where facilities differ for reasons uncorrelated with overall quality — candidate sites for rule reform, since they generate inspection burden and citations without predicting anything.
4. The Diamond Reading: Wisdom as Substantial Compliance [ANALOGY]
The Diamond Model’s axes: stability/chaos, give/take, short/long term (an information-density gradient), with scale-shell ordering as the fourth. Its wisdom criterion: the least destructive sufficient word — the smallest intervention that accomplishes the necessary update.
Fiene’s own field carries a set of polemics he states explicitly: do-no-harm vs. do-good, gatekeeper vs. enabler, risk vs. performance, full vs. substantial compliance. Map them:
Do-no-harm is the stability pole; do-good is the give pole. Licensing (health and safety floors) holds the stability axis; quality systems (QRIS, professional development) operate on the give axis. Fiene’s insistence that these are different measurement systems — nominal versus ordinal, ceiling versus no ceiling — is the observation that the axes are orthogonal. You cannot buy give with stability enforcement. The curve goes flat.
The ceiling effect is the information-density gradient made visible. Short-term axis: this inspection, this citation, this rule. Long-term axis: the facility’s developmental trajectory. Enforcement past substantial compliance spends long-term resources (staff attention, program budget, goodwill) to purchase short-term bits that carry no information — trailing zeros, again. The compliance-outcome curve bends because the information density of additional enforcement goes to zero while its cost does not.
Substantial compliance IS the least destructive sufficient word. This is the sentence the whole paper exists to earn. Fiene’s empirical finding — facilities at 100% compliance underperforming facilities at substantial compliance — is the wisdom-ordering policy validated on decades of licensing data. The regulator who demands the last 2% is not maximally safe; they are speaking a longer word than the update requires, and the excess length is destructive. The gatekeeper who cannot stop gatekeeping becomes the hazard.
Computed toy [COMPUTED]. Under a resource-tradeoff model (compliance cost superlinear near 1.0, remainder invested in program quality), the outcome-optimal compliance target lands at 0.88, and outcome(0.93) = 0.775 strictly beats outcome(1.00) = 0.720. The toy does not prove Fiene’s empirical curve — his data does that — but it shows the mechanism requires nothing exotic: any system where the marginal cost of the final bits exceeds their marginal information will produce his ceiling.
[CONJ] The regulator’s per-facility decision — abbreviated pass, full inspection, technical assistance, enforcement action — is a branch choice on the three axes: how much stability is actually at risk (risk quotient), whether the posture is give (assistance) or take (sanction), and which time horizon the intervention prices. Fiene’s 2×2 matrices are two-axis slices of this diamond; the risk assessment matrix fixes give/take and works stability × probability, the compliance-decision matrix fixes stability and works certainty × response. A full three-axis instrument — a compass parser for regulators — is a buildable object, not a metaphor.
5. Predictions and Falsifiers [PENDING]
P1. On any real state licensing dataset, rank rules by single-pass entropy, then by phi. Prediction: published key indicator sets for that state fall inside the top-half entropy stratum (the odd part). Falsifier: a validated key indicator with near-zero compliance variance.
P2. Delta-triggered full inspections (predicted shortfall from an indicator-trained model) will beat that state’s current targeting rule on violator recall at matched budget, or match it — because the state’s rule already approximates a TDIL controller. Falsifier: uniform random targeting outperforming delta-triggering on held-out data.
P3. Rules with high entropy but persistently low phi across multiple states are enriched for rules later amended or dropped in regulatory review. Falsifier: reform activity uncorrelated with the high-entropy/low-phi stratum.
6. What We Are and Are Not Claiming
Not claiming: that Fiene used, needed, or anticipated this vocabulary; that the analogies are isomorphisms; that a toy simulation validates fifty years of field research. The arrows of credit point one way — he found the architecture in the data before the architecture had a general name.
Claiming: that key indicators are information-theoretic channel selection [COMPUTED, on simulation]; that differential monitoring is a delta-triggered predictive controller [COMPUTED, on simulation]; that the ceiling effect is what an information-density gradient looks like when you price it [COMPUTED, toy mechanism]; and that “substantial beats full” is the wisdom-ordering policy of the Diamond Model, discovered independently, in the one domain civilized enough to write it into monitoring policy.
7. E-Tree Treatment of the Compliance Corpus [COMPUTED]
The licensing corpus is a bundle of short binary words. Three E-Tree techniques transfer directly — compliance data is not spatial, but the E-Tree’s exportable content was never geometry; it was locality-preserving keys, per-lane aggregates, complement duality, and odd-part storage. All results below reproduce from the companion harnesses, seed 27.
E1 — Pruned traversal (carry-accelerator pattern). Morton-interleave the six key-indicator bits with two risk-tier bits into a locality-preserving key; sort facilities on it; build an implicit binary tree whose node aggregate is max predicted shortfall in the subtree. The differential-monitoring query — every facility above the full-inspection trigger — then prunes whole subtrees whose aggregate falls below threshold. Measured: 665 facilities returned, exact match with brute-force scan, 91 of 255 tree nodes visited, 66.4% of item scans pruned. The pruning is strong for the same reason the spatial accelerator’s was: the sort key is the predictor, so hot facilities cluster in key order.
E2 — Storage via odd-part strip, with an honest failure first. The naive attempt lost to a bit-packed dense matrix (0.61:1) — at 50 high-entropy rules with 2-byte exception records, the word is already near its odd core and exceptions cost more than the bits they replace. Compression exists only where redundancy exists; the toy world had too little. Under realistic rule-book structure (hundreds of rules, ~80% near-universal compliance — i.e., mostly zero-entropy bits) and bit-tight encoding — corpus template for the even part, per-Morton-bucket majority templates plus XOR exception lists for the odd part — the ratio recovers: 5.8:1 at 50 rules, ~3:1 sustained at 200–800 rules. Same mechanism as the 47:1 spatial result, smaller words: store the irreducible core once, exceptions sparsely, derive everything else.
E3 — Complement duality + popcount. Pack each facility’s word into a uint64. Compliance score = popcount(word), measured 48× faster than row summation over the byte matrix; violation count = M − popcount, obtained by complement with zero additional storage. The violation record is the complement traversal of the compliance record — the sibling-thread duality, verbatim.
[CONJ] At real-registry scale (tens of thousands of facilities × 400+ rules × monthly snapshots), the three combine: odd-part strip decides what to store, Morton order decides where, node aggregates decide how to search, duality decides what never to duplicate. A state licensing database is a small E-Tree problem wearing a filing cabinet.
8. References [REF]
- Fiene, R. — Regulatory Compliance and Monitoring Systems, 4th ed. (NARA/RIKI). The textbook; Ch. 5 covers differential monitoring, key indicators, risk assessment.
- Fiene, R. (2024) — “The Regulatory Compliance Matrices: Risk, Compliance, and Licensing Decision Making.” The 2×2 standardization.
- Fiene, R. (2025) — “Regulatory Science, Risk Assessment, Predictor Rules and the Regulatory Compliance Scale,” Medium. Current synthesis of risk assessment vs. predictor rules.
- Fiene, R. — Theory of Regulatory Compliance papers (SSRN; RIKI technical research notes). Ceiling effect, diminishing returns, dichotomization of licensing data.
- RIKI Institute — rikinstitute.com; Fiene’s Medium — @rickfiene.
- WodansEye Research Series — TDIL architecture papers; Odd-Part Theorem and Takagi Form of the Switching Law; Diamond Model.
Companion harnesses: fiene_tdil_check.py (T1–T4), fiene_etree_accel.py (E1–E3), e2b_scale.py (E2 scale sweep) — 2,000 facilities, seed 27; every numbered result reproduces from a single run of each file.
Appendix A — Fiene TDIL Check PY
"""
fiene_tdil_check.py — WodansEye harness
Fiene's regulatory methodologies re-derived as TDIL / odd-part operations.
Claims tested:
T1 [COMPUTED] Key indicators (Fiene: high-phi rules) = high-mutual-information
channels. Sparse real channels reconstruct the full compliance vector
(TDIL virtual channel synthesis).
T2 [COMPUTED] Differential monitoring (delta-triggered full inspection)
beats uniform monitoring at fixed inspection budget.
T3 [COMPUTED] Odd-part extraction: zero-entropy rules are the trailing
zeros of the compliance word; the discriminative core (odd part) is
small and contains all key indicators.
T4 [COMPUTED] Ceiling effect: under a resource-tradeoff model, substantial
compliance yields better child outcomes than full compliance
(Fiene's diminishing-returns / Theory of Regulatory Compliance).
"""
import numpy as np
rng = np.random.default_rng(27) # gold-7 kernel salute
N_FAC, N_RULES = 2000, 50
# --- world model -----------------------------------------------------------
# latent quality q in [0,1]; rules have difficulty d_j and risk weight w_j
q = rng.beta(4, 2, N_FAC) # most facilities decent
diff = rng.uniform(0.05, 0.9, N_RULES) # rule difficulty
risk_w = rng.choice([1, 2, 4, 8], N_RULES, # Fiene risk quotient (severity x likelihood)
p=[.4, .3, .2, .1])
noise = rng.normal(0, 0.12, (N_FAC, N_RULES))
C = (q[:, None] + noise > diff[None, :]).astype(int) # compliance matrix, nominal 1/0
full_score = C.mean(1)
SUBSTANTIAL = 0.90 # Fiene: substantial-compliance cut
substantial = (full_score >= SUBSTANTIAL).astype(int)
# true violators = facilities with any high-risk rule out of compliance
violator = ((C == 0) & (risk_w[None, :] >= 8)).any(1).astype(int)
# --- T1: key indicators as high-MI channels --------------------------------
def phi(x, y):
a = np.mean((x == 1) & (y == 1)); b = np.mean((x == 1) & (y == 0))
c = np.mean((x == 0) & (y == 1)); d = np.mean((x == 0) & (y == 0))
den = np.sqrt((a + b) * (c + d) * (a + c) * (b + d))
return 0.0 if den == 0 else (a * d - b * c) / den
phis = np.array([phi(C[:, j], substantial) for j in range(N_RULES)])
K = 6 # abbreviated inspection size
key_idx = np.argsort(-phis)[:K] # Fiene key indicators
# TDIL synthesis: predict every rule bit from the K real channels
# effective value = observed key bits; predicted value = virtual channels
from numpy.linalg import lstsq
X = np.hstack([C[:, key_idx], np.ones((N_FAC, 1))])
train = rng.random(N_FAC) < 0.5
W, *_ = lstsq(X[train], C[train], rcond=None)
V = np.clip(X[~train] @ W, 0, 1) # virtual compliance channels
recon_acc = ((V > 0.5).astype(int) == C[~train]).mean()
pred_sub = ((V.mean(1) >= SUBSTANTIAL).astype(int) == substantial[~train]).mean()
# --- T2: differential vs uniform monitoring at fixed budget ----------------
delta = np.abs(V.mean(1) - SUBSTANTIAL) * -1 + (SUBSTANTIAL - V.mean(1)) # low predicted -> high delta
delta = SUBSTANTIAL - V.mean(1) # anomaly score: predicted shortfall
budget = int(0.20 * (~train).sum()) # full inspections for 20% of facilities
test_viol = violator[~train]
diff_pick = np.argsort(-delta)[:budget] # delta-triggered (TDIL controller)
unif_pick = rng.choice(len(delta), budget, replace=False)
diff_catch = test_viol[diff_pick].mean()
unif_catch = test_viol[unif_pick].mean()
diff_recall = test_viol[diff_pick].sum() / max(test_viol.sum(), 1)
unif_recall = test_viol[unif_pick].sum() / max(test_viol.sum(), 1)
# --- T3: odd part of the compliance word ------------------------------------
# entropy per rule; zero/near-zero entropy rules = trailing zeros (strip them)
p1 = C.mean(0)
H = -(np.where(p1 > 0, p1 * np.log2(p1), 0) +
np.where(p1 < 1, (1 - p1) * np.log2(1 - p1), 0))
STRIP = 0.30 # bits/rule threshold
odd_part = np.where(H > STRIP)[0] # irreducible discriminative core
keys_in_odd = np.isin(key_idx, odd_part).all()
# --- T4: ceiling effect / substantial beats full ----------------------------
# resource tradeoff: pushing compliance above substantial consumes quality budget
R = 1.0 # facility resource budget
def outcome(compliance_target):
# cost of compliance grows superlinearly near 1.0 (diminishing returns)
cost = 0.55 * compliance_target ** 4
quality_invest = np.maximum(R - cost, 0)
return 0.6 * np.minimum(compliance_target, SUBSTANTIAL) + 0.4 * quality_invest
targets = np.linspace(0.5, 1.0, 26)
outs = np.array([outcome(t) for t in targets])
best_t = targets[np.argmax(outs)]
o_sub, o_full = outcome(0.93), outcome(1.0)
print(f"T1 key indicators (phi-selected): rules {sorted(key_idx.tolist())}")
print(f"T1 TDIL reconstruction of 50-rule vector from {K} channels: {recon_acc:.3f} per-bit acc")
print(f"T1 substantial-compliance classification via virtual channels: {pred_sub:.3f}")
print(f"T2 fixed budget = 20% full inspections")
print(f"T2 delta-triggered precision {diff_catch:.3f} vs uniform {unif_catch:.3f}")
print(f"T2 delta-triggered recall {diff_recall:.3f} vs uniform {unif_recall:.3f}")
print(f"T3 odd part: {len(odd_part)}/{N_RULES} rules survive entropy strip; "
f"all key indicators inside odd part: {keys_in_odd}")
print(f"T4 optimal compliance target {best_t:.2f}; outcome(0.93)={o_sub:.3f} "
f"> outcome(1.00)={o_full:.3f}: {o_sub > o_full}")Appendix B — Fiene Etree Acceleration PY
"""
fiene_etree_accel.py — WodansEye harness, part 2
E-Tree techniques applied to regulatory compliance collections.
Claims tested:
E1 [COMPUTED] Morton-keyed implicit BVH with max-delta node aggregates
prunes the delta-threshold query (find facilities needing full
inspection) vs brute-force scan.
E2 [COMPUTED] Storage: odd-part strip + bucket-template XOR encoding
reduces bytes vs bit-packed dense matrix. Honest ratio reported.
E3 [COMPUTED] Complement duality + packed popcount: compliance scoring
on uint64 words beats per-byte summation.
"""
import numpy as np, time
rng = np.random.default_rng(27)
N_FAC, N_RULES = 2000, 50
# --- same world as fiene_tdil_check.py -------------------------------------
q = rng.beta(4, 2, N_FAC)
diff = rng.uniform(0.05, 0.9, N_RULES)
risk_w = rng.choice([1, 2, 4, 8], N_RULES, p=[.4, .3, .2, .1])
C = (q[:, None] + rng.normal(0, 0.12, (N_FAC, N_RULES)) > diff[None, :]).astype(np.uint8)
full = C.mean(1); SUB = 0.90
substantial = (full >= SUB).astype(int)
def phi_col(j):
x, y = C[:, j], substantial
a = np.mean((x==1)&(y==1)); b = np.mean((x==1)&(y==0))
c = np.mean((x==0)&(y==1)); d = np.mean((x==0)&(y==0))
den = np.sqrt((a+b)*(c+d)*(a+c)*(b+d))
return 0.0 if den == 0 else (a*d - b*c)/den
key_idx = np.argsort([-phi_col(j) for j in range(N_RULES)])[:6]
risk_tier = np.digitize((C[:, risk_w >= 4] == 0).sum(1), [1, 3]) # 0/1/2
# =========================================================================
# E1: Morton key + implicit BVH with node aggregates (carry-accelerator style)
# =========================================================================
# Key = interleave(6 key-indicator bits, 2 risk-tier bits): facilities with
# similar abbreviated-inspection signatures land adjacent in sort order.
kbits = C[:, key_idx] # (N,6)
def morton(kb, rt):
key = 0
for i in range(6):
key |= int(kb[i]) << (2*i) # even lanes: indicator bits
key |= (int(rt) & 1) << 1 | ((int(rt) >> 1) & 1) << 3 # odd lanes: risk
return key
keys = np.array([morton(kbits[i], risk_tier[i]) for i in range(N_FAC)])
order = np.argsort(keys, kind="stable")
# delta from part-1 style prediction: shortfall of bucket-mean compliance
bucket_id = keys
bmean = {b: full[bucket_id == b].mean() for b in np.unique(bucket_id)}
delta = np.array([SUB - bmean[b] for b in bucket_id]) # predicted shortfall
dsort = delta[order]
# implicit binary tree over the sorted array, node aggregate = max delta
LEAF = 16
n_leaves = int(np.ceil(N_FAC / LEAF))
size = 1
while size < n_leaves: size *= 2
agg = np.full(2*size, -np.inf)
for L in range(n_leaves):
agg[size+L] = dsort[L*LEAF:(L+1)*LEAF].max()
for i in range(size-1, 0, -1):
agg[i] = max(agg[2*i], agg[2*i+1])
visited = 0
def query(node, thr, out):
global visited
visited += 1
if agg[node] < thr: return # prune whole subtree
if node >= size:
L = node - size
seg = slice(L*LEAF, (L+1)*LEAF)
idx = np.where(dsort[seg] >= thr)[0] + L*LEAF
out.extend(order[idx].tolist()); return
query(2*node, thr, out); query(2*node+1, thr, out)
THR = np.quantile(delta, 0.90) # top-decile shortfall
hits = []; query(1, THR, hits)
brute = set(np.where(delta >= THR)[0].tolist())
assert set(hits) == brute, "BVH query mismatch"
touched_leaf_items = sum(LEAF for _ in range(0)) # accounted below
leaves_touched = sum(1 for L in range(n_leaves)
if dsort[L*LEAF:(L+1)*LEAF].max() >= THR)
items_examined = leaves_touched * LEAF
print(f"E1 query: delta >= {THR:.3f} -> {len(hits)} facilities (matches brute force)")
print(f"E1 nodes visited {visited}/{2*size-1}; items examined {items_examined}/{N_FAC} "
f"({100*(1-items_examined/N_FAC):.1f}% of scans pruned)")
# =========================================================================
# E2: storage — odd-part strip + bucket-template XOR exceptions
# =========================================================================
dense_bytes = N_FAC * N_RULES // 8 # bit-packed baseline
p1 = C.mean(0)
H = -(np.where(p1>0, p1*np.log2(p1), 0) + np.where(p1<1, (1-p1)*np.log2(1-p1), 0))
odd = np.where(H > 0.30)[0] # entropy-bearing rules
even = np.setdiff1d(np.arange(N_RULES), odd)
# even part: one corpus template + global exception list (facility,rule)
even_template = (p1[even] > 0.5).astype(np.uint8)
even_exc = int((C[:, even] != even_template[None, :]).sum())
# odd part: per-Morton-bucket majority template + per-facility XOR exceptions
odd_exc, n_buckets = 0, 0
for b in np.unique(bucket_id):
rows = C[bucket_id == b][:, odd]
tmpl = (rows.mean(0) > 0.5).astype(np.uint8)
odd_exc += int((rows != tmpl[None, :]).sum()); n_buckets += 1
BYTES_PER_EXC = 2 # (facility-local id + 6-bit rule index) packed, conservative
etree_bytes = (n_buckets * len(odd) // 8 + n_buckets # bucket templates + counts
+ len(even) // 8 + 1 # corpus template
+ (even_exc + odd_exc) * BYTES_PER_EXC # exception lists
+ N_FAC * 1) # per-facility bucket key
ratio = dense_bytes / etree_bytes
print(f"E2 dense bit-packed: {dense_bytes} B | e-tree encoding: {etree_bytes} B "
f"({ratio:.2f}:1) [odd part {len(odd)}/{N_RULES} rules; "
f"exceptions: even {even_exc}, odd {odd_exc}]")
# =========================================================================
# E3: complement duality + popcount scoring on packed words
# =========================================================================
packed = np.zeros(N_FAC, dtype=np.uint64)
for j in range(N_RULES):
packed |= C[:, j].astype(np.uint64) << np.uint64(j)
t0 = time.perf_counter()
for _ in range(200): s1 = C.sum(1)
t_naive = time.perf_counter() - t0
t0 = time.perf_counter()
for _ in range(200):
s2 = np.bitwise_count(packed) # popcount per facility
t_pop = time.perf_counter() - t0
assert (s1 == s2).all()
# duality: violation count = 50 - popcount(word); no second matrix stored
viol = N_RULES - s2
print(f"E3 scoring 2000 facilities x200 reps: naive {t_naive*1000:.1f} ms, "
f"popcount {t_pop*1000:.1f} ms ({t_naive/t_pop:.1f}x); "
f"violation counts via complement, zero extra storage")Appendix C — E2b Scale
"""E2b: storage ratio vs rule-book size. Tight bit-level encoding:
odd part bucket-XOR (6-bit-capped indices -> ceil(log2) bits), even part
corpus template + global exceptions. Honest accounting in bits."""
import numpy as np
rng = np.random.default_rng(27)
N = 2000
def world(M, frac_lowH=0.8):
q = rng.beta(4,2,N)
# frac_lowH of rules are near-universal (low entropy), rest discriminative
n_low = int(M*frac_lowH)
diff = np.concatenate([rng.uniform(0.02,0.15,n_low), rng.uniform(0.3,0.85,M-n_low)])
C = (q[:,None] + rng.normal(0,0.12,(N,M)) > diff[None,:]).astype(np.uint8)
return C
def encode_bits(C):
N,M = C.shape
p1 = C.mean(0)
H = -(np.where(p1>0,p1*np.log2(p1),0)+np.where(p1<1,(1-p1)*np.log2(1-p1),0))
odd = np.where(H>0.30)[0]; even = np.setdiff1d(np.arange(M),odd)
# key = top-6 variance rules within odd part (indicator proxy)
key = odd[np.argsort(-H[odd])[:6]] if len(odd)>=6 else odd
bucket = (C[:,key] * (1<<np.arange(len(key)))).sum(1)
ridx_bits = max(1,int(np.ceil(np.log2(max(len(odd),2)))))
bits = 0
# even: corpus template + exceptions (11-bit fac id + log2(M) rule id)
tmpl_e = (p1[even]>0.5).astype(np.uint8)
exc_e = int((C[:,even]!=tmpl_e[None,:]).sum())
bits += len(even) + exc_e*(11+max(1,int(np.ceil(np.log2(max(len(even),2))))))
# odd: per-bucket template + per-facility exception list (count + indices)
for b in np.unique(bucket):
rows = C[bucket==b][:,odd]
t = (rows.mean(0)>0.5).astype(np.uint8)
exc = (rows!=t[None,:]).sum(1)
bits += len(odd) # bucket template
bits += int((np.ceil(np.log2(len(odd)+1)) + exc*ridx_bits).sum())
return bits, len(odd), len(even)
for M in [50, 200, 400, 800]:
C = world(M)
dense = N*M
et, no, ne = encode_bits(C)
print(f"M={M:4d} dense {dense/8/1024:7.1f} KB e-tree {et/8/1024:7.1f} KB "
f"ratio {dense/et:5.2f}:1 (odd part {no}, even {ne})")
