Documentation
¶
Overview ¶
Package memq is the agent-facing MEMORY-OPERATION ALGEBRA — the substrate that lets an agent (or a plugin, a driver, or an operator) author its OWN memory strategy instead of the kernel hard-coding one. The slogan is "build SQL, not a specific query": fak already ships several SPECIFIC memory operations welded into Go — recall.Recall (retrieve a working set), recall.Dream (consolidate/clean a core image), recall.RequestContextChange (tombstone one page), contextq.Query (materialize a render plan). Each is one fixed pipeline. memq is the small, composable language those five pipelines are five SENTENCES of, plus the planner and executor that run an authored sentence — so a sixth strategy nobody anticipated is a data value an agent emits, not a kernel edit.
The model ¶
- A Cell is one addressable unit of memory (a recall.Page, an in-memory note, a derived disposition) with TYPED attributes: role, kind, durability class, sealed/tombstoned flags, size, digest, references, and an OPEN Attrs bag (the same forward-compatible posture as abi.Verdict.Meta). Backends supply cells.
- A Pred is a SERIALIZABLE predicate expression (and/or/not/eq/ne/lt/le/gt/ge/ match) over a cell's fields — the WHERE clause, authorable as JSON, never a Go closure, so an agent or an MCP client can write one.
- A Query is an ordered pipeline of Ops: scan | filter | rank | limit | budget (the pure SELECT side) and dedup | render | tombstone | consolidate | reclassify | prune (the EFFECT side). The pipeline threads a working set of cells.
- Plan/Explain renders the pipeline (and its per-step cell counts) WITHOUT executing — the "step through this before you run it" surface.
- Run executes the pipeline against a Backend.
The safety posture (inherited from the rest of the kernel, enforced + tested) ¶
- There is NO hard-delete operator. The strongest forgetting is `tombstone` (negative-only: future recall skips the cell, the bytes and the row survive for audit — recall.RequestContextChange) and `prune` (unreferenced storage GC only). This mirrors fak's negative-only/expire-by-default stance (CONTEXT-IS-NOT-MEMORY.md).
- Effects default to PROPOSED, not applied. Run mutates a backend only when the caller grants an explicit Caps for that effect; without caps every mutation is reported as a proposal the operator can inspect first. Fail-closed: the default is don't-touch.
- render and consolidate NEVER read a sealed cell's bytes — Materialize routes every page-in through the backend's trust gate (recall's quarantine re-screen), so a poisoned span can never be rendered into context or folded into a summary.
- reclassify can never PROMOTE a cell to `durable`; it may only hold or lower a class (expire-by-default; promotion must be earned, which this rung does not grant). Same fail-closed instinct as recall's promotion gate.
- The whole pipeline is deterministic: a fixed (query, backend cells) yields a byte-identical plan and result (no RNG, clock, or map-iteration dependence) — witnessed in proofs_witness_test.go.
Honest scope (rung 1) ¶
render, tombstone, prune, and reclassify are REAL: render pages bytes in through the gate; tombstone applies through recall.RequestContextChange (with caps) and persists; prune reclaims unreferenced storage on a backend that supports it; reclassify — with a Caps grant and a backend implementing Reclassifier — persists a LOWERED durability class back to the store and mints a demotion audit record on the promotion ledger (#4147). It stays demote-only even under caps: a promotion is still refused, capped at the current class, and a backend without the seam is proposal-only (the safe floor). consolidate produces a REAL derived artifact (a deterministic extractive summary the agent can render into context) but does NOT yet write that disposition back to durable store — the consolidate durable write-back is the named rung-2 follow-on (#3906), which SHARES the reclassify write-back seam. memq COMPLEMENTS the existing recall.Dream (which still owns the trust-gate reseal half of a sleep pass); it does not replace it.
Index ¶
- Constants
- Variables
- func Digest(b []byte) string
- func GateEphemeralPromotion(text string, reclass string) ctxmmu.EphemeralGateOutcome
- func IsEffect(kind string) bool
- func IsMutation(kind string) bool
- func NormConsent(s string) string
- func NormDurability(s string) string
- func NormProducer(s string) string
- func NormReclassification(s string) string
- func ProbeAtWrite(ctx context.Context, raw []byte, verifier recall.ArtifactVerifier) ([]byte, []recall.ArtifactFinding)
- func Register(d Driver)
- func RenderNotesDigest(storeDir string, indexOnly bool, maxBytes int) string
- func SameSubject(a, b PromotionRecord) bool
- func ValidConflictOutcome(o ConflictOutcome) bool
- func Validate(q Query) error
- type AdmissionVerdict
- type Backend
- type Caps
- type Cell
- type CodexBackend
- type ConflictDecision
- type ConflictOutcome
- type Driver
- type Effect
- type Explanation
- type IndexOverflow
- type MemStore
- func (m *MemStore) Add(role, kind, durability string, body []byte, sealed bool) Cell
- func (m *MemStore) AddIfDurable(role, kind, durability string, body []byte, sealed bool, reclass string) (Cell, error)
- func (m *MemStore) AddOrphanBlob(body []byte) string
- func (m *MemStore) AddPromoted(role, kind, durability string, body []byte, sealed bool, meta PromotionMeta) Cell
- func (m *MemStore) AddPromotedIfDurable(role, kind, durability string, body []byte, sealed bool, meta PromotionMeta, ...) (Cell, error)
- func (m *MemStore) ApplyStarveUpdates(updates []StarveUpdate) int
- func (m *MemStore) Cells(_ context.Context) ([]Cell, error)
- func (m *MemStore) Materialize(_ context.Context, id string) ([]byte, error)
- func (m *MemStore) Promotions() *PromotionLedger
- func (m *MemStore) Prune(_ context.Context, apply bool) (int, int64, error)
- func (m *MemStore) Reclassify(_ context.Context, id, newClass, by string) (bool, error)
- func (m *MemStore) Tombstone(_ context.Context, id, _, _ string) (bool, error)
- type NearDupPair
- type NotesBackend
- func (b *NotesBackend) Cells(_ context.Context) ([]Cell, error)
- func (b *NotesBackend) Dir() string
- func (b *NotesBackend) HedgeReason(ctx context.Context, id string) (string, error)
- func (b *NotesBackend) Materialize(ctx context.Context, id string) ([]byte, error)
- func (b *NotesBackend) Verify(ctx context.Context, id string) ([]recall.ArtifactFinding, error)
- func (b *NotesBackend) WithVerifier(v recall.ArtifactVerifier) *NotesBackend
- type Op
- type OverflowEntry
- type Params
- type Plan
- type PlanStep
- type Pred
- type PromotionLedger
- func (l *PromotionLedger) All() []PromotionRecord
- func (l *PromotionLedger) Conflicts(cellID string) []ConflictDecision
- func (l *PromotionLedger) Explain(cellID string) Explanation
- func (l *PromotionLedger) For(cellID string) (recs []PromotionRecord, ok bool)
- func (l *PromotionLedger) Latest(cellID string) (PromotionRecord, bool)
- func (l *PromotionLedger) Record(rec PromotionRecord)
- func (l *PromotionLedger) RecordReclassify(cellID string, span SourceSpan, from, to, by string) bool
- func (l *PromotionLedger) Timeline(cellID string) memview.Timeline
- type PromotionMeta
- type PromotionRecord
- type Pruner
- type Query
- type RecallBackend
- type Reclassifier
- type Refusal
- type RenderItem
- type Result
- type SourceSpan
- type StarveReport
- type StarveUpdate
- type Stats
- type StepTrace
- type Tombstoner
Constants ¶
const ( // AdmitOK is the verdict for a candidate write that passes every structural rule. AdmitOK = "ok" // RefuseOversizeVerbatim refuses a durable write whose body exceeds the // single-fact byte bound — a document-sized verbatim copy (the #3006 74 KiB // skill body), never a distilled fact. RefuseOversizeVerbatim = "oversize_verbatim" // RefuseTransientError refuses a durable write whose fact is a transient // environment failure (a connection reset, a timeout, a rate-limit wall) — a // one-off event, not a durable invariant. RefuseTransientError = "transient_error" // RefuseNegativeToolClaim refuses a durable write whose fact is a negative // tool-INVOCATION claim (a call that failed / exited nonzero / produced no // output this run) — a session-transient narrative, not a durable fact. RefuseNegativeToolClaim = "negative_tool_claim" // RefuseOneOffNarrative refuses a durable write whose fact is a one-off session // narrative — a thin event line anchored to THIS run/session/turn by a temporal // deictic ("this session", "just now", "earlier today"). A durable fact is a // timeless generalization and never needs to bind itself to the current run; the // bound narrative is the fourth "Do NOT capture" class the Hermes prose list names // but cannot enforce (#2836). RefuseOneOffNarrative = "one_off_narrative" )
Deny-by-structure memory-write adjudication (#2912). Hermes curates durable memory by asking the model to honor a prose "Do NOT capture" list in the review prompt (environment failures, negative tool claims, transient errors) — enforcement is hope-the-model-complies, and #3006 is the failure: a 74 KiB skill body copied verbatim into a single memory entry. A list in a prompt is not an invariant; a distracted or injected model writes junk to durable memory.
fak adjudicates a tool call deny-by-structure; a durable memory write is just another governed syscall. This is that structural arm: a candidate durable write is judged by SHAPE — its size, and the lexical signature of the three junk classes the Hermes list names but cannot enforce (a transient environment error, a negative tool-INVOCATION claim, a one-off session narrative bound to this run) — and refused with a reason from a closed vocabulary BEFORE it reaches storage. No prompt, no model call, no override string: structure decides, so an injected or distracted model cannot talk its way past it (unlike the ephemeral gate in ephemeral_gate.go, whose situational refusal a caller MAY override with an explicit reclassification). The one-off-narrative arm is issue #2836's delta over #2912.
The rules are deliberately conservative — a durable fact is a distilled generalization, so the refusals target the shapes that are junk BY CONSTRUCTION (a document-sized verbatim blob; a bare event line reporting a transient failure or a failed tool invocation) and leave a legitimate distilled fact untouched even when it MENTIONS a retry, a missing binary, or a rate-limit header. Over-refusing an honest report is the named risk (the issue's confusion note); the rules err toward under-refusing. In particular the negative-tool arm targets a failed INVOCATION ("failed to run X", "the call exited nonzero") — a session-transient narrative — NOT a durable CAPABILITY fact ("grep is not on PATH on this host"), which is exactly the kind of environment fact worth keeping. Likewise the one-off arm targets a fact BOUND to this run by a temporal deictic ("this session", "just now") — the fourth "Do NOT capture" class — NOT a durable fact that merely names a session/run/turn ("the session token expires in 30m").
const ( // CodexProvenance is the witness/provenance label: this is external, generated, // untrusted state — never a fak- or team-authored rule. CodexProvenance = "external/untrusted" // KindCodexMemory tags an ordinary generated memory file (<home>/memories). KindCodexMemory = "codex-memory" // KindCodexChronicle tags a Chronicle-origin memory (screen-derived; higher risk). KindCodexChronicle = "codex-chronicle" )
Provenance / attribute vocabulary stamped on every Codex cell. These are plain strings (memq carries no enum for external provenance) so a Pred can select on attr:provenance / attr:source / kind without the core knowing about Codex.
const ( ReclassifyNone = "none" ReclassifyExplicitConsent = "explicit_consent" ReclassifyUserConfirmed = "user_confirmed" ReclassifyEstablishedPattern = "established_pattern" )
EphemeralReclassification is memq's plain-string mirror of ctxmmu.Reclassification — memq.go's existing pattern for the durability/consent vocabularies (plain strings, normalized fail-closed) rather than importing the mechanism package's enum into the public API. NormReclassification maps between the two.
const ( DurabilityTurn = "turn" DurabilitySession = "session" DurabilityBounded = "bounded" DurabilityDurable = "durable" )
Durability classes — the temporal axis from CONTEXT-IS-NOT-MEMORY.md. memq mirrors the recall/ctxmmu vocabulary as plain strings (it does not import the mechanism packages) and normalizes any missing/unknown class to the shortest-lived one.
const ( OpScan = "scan" // source: (re)load every cell from the backend OpFilter = "filter" // keep cells matching Pred (WHERE) OpRank = "rank" // sort by By (ORDER BY) OpLimit = "limit" // keep the first K (LIMIT) OpBudget = "budget" // keep the prefix whose cumulative Bytes <= Bytes OpDedup = "dedup" // collapse digest-identical cells to one (read-side recall dedup; #2506) OpNarrow = "narrow" // shrink kept cells' width to a per-cell byte cap (opt-in 2nd compression axis; ThinK #4019) OpRender = "render" // materialize the set into context (read-only page-in via the gate) OpTombstone = "tombstone" // negative-only suppression (recall.RequestContextChange) OpConsolidate = "consolidate" // fold the set into one derived extractive disposition OpReclassify = "reclassify" // change durability class (never promotes to durable) OpPrune = "prune" // reclaim unreferenced storage (GC); no model-visible effect OpExempt = "exempt" // carve the top-K most-divergent cells out of the set before a lossy fold, kept bit-exact (MiniCache; #4018) )
Op kinds. The pure SELECT side transforms the working set; the EFFECT side records (and, with caps, applies) a mutation or materialization.
const ( PredTrue = "true" PredAnd = "and" PredOr = "or" PredNot = "not" PredEq = "eq" PredNe = "ne" PredLt = "lt" PredLe = "le" PredGt = "gt" PredGe = "ge" PredMatch = "match" // token overlap of Value against (role + descriptor) > 0 PredHasRef = "hasref" // Value is a member of the cell's Refs (the [[wikilink]] edge set); "references <id>" )
Pred ops. Comparison ops resolve a field; the boolean ops compose sub-predicates.
const ( RankRelevance = "relevance" // token overlap with the query Intent (the recall ranker) RankBytes = "bytes" // size RankStep = "step" // ordinal position RankDurability = "durability" // shortest-lived first (asc) / longest-lived first (desc) RankRefcount = "refcount" // graph in-degree — how many other cells reference this one (desc = most backlinks first) // RankRelevanceSpan is the neighborhood-pooled relevance variant (#4014, SnapKV): // each cell's intent-overlap score is summed over an odd, centered window of its // step-adjacent neighbors before ranking, so a coherent multi-cell span outranks an // isolated one-cell spike. Opt-in only — it requires Op.Window (odd, >= 1); window 1 // is the identity kernel and reproduces RankRelevance's ordering exactly. RankRelevanceSpan = "relevance_span" RankDivergence = "divergence" // simhash dissimilarity (1 - cosine) to the working set's consolidation centroid (desc = most divergent first; #4018) )
Rank keys.
const ( ScoreAggMean = "mean" // fold to the per-intent average (the diluting default) ScoreAggAmax = "amax" // fold to the per-intent maximum (any one consumer's need keeps the cell) ScoreAggSum = "sum" // fold to the per-intent total (mean without the /N normalization) )
Score-aggregation ops for multi-intent recall (#4020 — the GQA group-reduce: N consumer intents share ONE retained set, so their per-intent relevance scores fold to one shared score per cell before the single top-k). The op choice is deliberately a knob, exposed exactly as upstream exposes it: mean dilutes a strong minority signal, amax over-keeps for any single consumer. Empty means ScoreAggMean.
const ( // NotesProvenance labels a note as the agent's own accumulated memory: curated, // but still a generated self-report — verified at read time, never taken on faith. NotesProvenance = "memory-store/self-report" // KindMemoryNote tags one MEMORY.md-indexed fact file. KindMemoryNote = "memory-note" )
Provenance / kind vocabulary stamped on every note cell, selectable via attr:provenance / kind without the core knowing about the store.
const ( ConsentExplicit = "explicit" // the user/operator directly asked for this to be remembered ConsentInferred = "inferred" // the producer inferred durability (e.g. tense/lexical prior, a driver default) ConsentUnknown = "unknown" // no consent signal was recorded; fails closed to the weakest claim )
Consent classes for a context->memory promotion — was the durable write something the user (or an equivalent authoritative producer) explicitly asked for, or did the system infer it was durable-worthy without being told? This is a closed, small vocabulary (mirroring the closed Durability* vocabulary above) so an explain surface can render it without free text.
const ( // WriteVerifyKey is the top-level frontmatter field the probe stamps. The read // side keys its hedge on this structure, not on the note's prose. WriteVerifyKey = "write_verify" // WriteVerifyDetailKey names the failing claim (unverified) or the probed // claims (verified) so a reader can see what was checked and why. WriteVerifyDetailKey = "write_verify_detail" // WriteVerified stamps a note whose every checkable claim verified at write. WriteVerified = "verified-at-write" // WriteUnverified stamps a note with at least one claim that did not verify. WriteUnverified = "unverified-at-write" )
Write-time artifact-claim probe (#2431). The notes backend re-verifies a note's concrete claims at READ time (Materialize / Verify); ProbeAtWrite is the WRITE arm — run before a memory note lands so a self-reported artifact (a commit SHA, a repo path, a CLI flag the session narrated) is probed against the live checkout AT THE DOOR, and the verdict is stamped into the note's frontmatter as trusted structure the memoryread grammar carries — never prose.
A failing claim does NOT block the write: the observation may still be useful, so the note lands, but stamped unverified-at-write with the failing claim named. A note whose every checkable claim verifies carries verified-at-write with the probe witness. Recall then renders an unverified-at-write note hedged by default (HedgeReason) — even when a later live re-check is inconclusive, the doubt recorded at the door survives to the table. This closes the loop the read gate opened: garbage is labeled at the door and re-checked at the table (#2077).
A note with no checkable claim gets no write verdict — nothing was probed, so nothing is stamped; the read path already hedges a claim-free note (empty findings), exactly as before.
const MaxDurableFactBytes = 16 << 10 // 16 KiB
MaxDurableFactBytes is the single-fact size bound. A durable memory entry is one distilled fact (a sentence or short paragraph); anything past this is a document-sized verbatim copy that belongs in docs/, not a memory cell. #3006's 74 KiB write is more than 4x over. The bound sits well above any real distilled fact (a few hundred bytes to low-single-KiB), so it refuses the copied-document shape without touching a legitimate long fact.
const MergeOnEvictKind = "merge_on_evict"
MergeOnEvictKind labels the audit Effect a budget op's opt-in merge-on-evict fold records (#4015): a below-budget cell folded into its nearest surviving cell rather than tail-dropped. It is an EFFECT kind, never a pipeline op — the fold is a modifier on OpBudget (Op.MergeFloor), and it is proposal-only (Applied stays false; like OpConsolidate, the durable write-back is rung 2). One Effect is recorded per fold, naming [survivor, evicted] so the provenance of each absorption is auditable.
const OverflowReason = "MEMORY_INDEX_OVERFLOW"
OverflowReason is the closed-vocabulary verdict name stamped on an over-budget memory index (#2430). It is a TYPED event, not a free-text note: a consumer keys off this exact string to route the overflow list into the compaction/write pass.
const SpanGroupAttr = "thread"
SpanGroupAttr is the OPEN attribute-bag key that scopes neighborhood pooling (RankRelevanceSpan, #4014): cells sharing Attrs["thread"] pool only against each other, so two interleaved conversations never smear relevance across the boundary. Cells without the attr share one default group — the common single-thread case.
const StarveAttr = "memq.starve"
StarveAttr is the Attrs key carrying a cell's consecutive-below-cutline pass counter. It rides the OPEN attribute bag (forward-compatible, drop-unknown), so no Cell schema change is needed and any backend that round-trips Attrs persists it for free.
const StarveReason = "MEMORY_STARVE_CREDIT"
StarveReason is the closed-vocabulary verdict name stamped when the credit retains a perennially-below-cutline cell (#4021). Like OverflowReason, it is a TYPED event a consumer keys off — never free text.
Variables ¶
var ErrEphemeralRefused = errors.New("memq: situational observation refused for durable memory; expire by default, promote only with explicit reclassification")
ErrEphemeralRefused is returned when a durable-targeted Add/AddPromoted is refused because the body is a situational (timestamp/current-step/mood-bound) observation with no explicit reclassification — issue #1592's write-time gate. memq core otherwise avoids importing a mechanism package (memq.go's own comment: "it does not import the mechanism packages"), but codexbackend.go already imports ctxmmu for ScreenBytes at read time; this is the same precedent applied at the write/promotion boundary, and internal/architest's tier ladder allows it (ctxmmu is tier 2, memq is tier 3 — a downward import).
var ErrSealed = errors.New("memq: cell sealed by the trust gate")
ErrSealed is returned by a Backend.Materialize that refuses a page-in because the cell is quarantined by the trust gate. memq core never imports a mechanism package; a backend translates its own seal error (e.g. recall.ErrSealed) into this so the executor can label the refusal without the coupling.
var ErrStale = errors.New("memq: recall artifact stale")
ErrStale is returned by a Backend.Materialize that refuses a page-in because recall re-verified a concrete artifact claim at read time and found it stale.
var ErrWriteRefused = errors.New("memq: durable memory write refused by structure")
ErrWriteRefused wraps a deny-by-structure refusal of a durable write (#2912). The verdict's closed-vocabulary Reason and its Detail are formatted into the error; a caller that needs the machine token can call AdjudicateMemoryWrite directly.
Functions ¶
func Digest ¶
Digest is the canonical content address (sha256 hex), the same scheme recall/blob use, so a memq-derived digest and a recall digest are interchangeable.
func GateEphemeralPromotion ¶ added in v0.37.0
func GateEphemeralPromotion(text string, reclass string) ctxmmu.EphemeralGateOutcome
GateEphemeralPromotion is the pre-check callers are expected to run before promoting raw observation text into durable memory (#1592, done condition item 3): it wraps ctxmmu.GateEphemeral so a memq caller need not import ctxmmu directly for the common case. It returns the underlying ctxmmu outcome unchanged (Allowed/Situational/Reason) so a caller can audit WHY a promotion was refused or allowed.
func IsMutation ¶
IsMutation reports whether an op kind mutates durable backend state (needs caps).
func NormConsent ¶ added in v0.37.0
NormConsent maps any consent string to the canonical vocabulary, failing closed to ConsentUnknown for a missing/unrecognized value — same posture as NormDurability.
func NormDurability ¶
NormDurability maps any class string to the canonical vocabulary, failing closed to turn for a missing/unknown/reserved value — the reader-side default of the expire-by-default posture.
func NormProducer ¶ added in v0.37.0
NormProducer defaults an empty producer to "unknown" so PromotionRecord.Producer is never silently blank — same fail-closed-to-a-named-value posture as NormDurability and NormConsent.
func NormReclassification ¶ added in v0.37.0
NormReclassification maps any reclassification string to the canonical vocabulary, failing closed to ReclassifyNone for a missing/unrecognized value — same posture as NormConsent/NormDurability: an unrecognized override is treated as no override.
func ProbeAtWrite ¶ added in v0.38.0
func ProbeAtWrite(ctx context.Context, raw []byte, verifier recall.ArtifactVerifier) ([]byte, []recall.ArtifactFinding)
ProbeAtWrite runs the artifact verifier over a memory note's concrete claims BEFORE it lands and returns the note with a write_verify frontmatter stamp plus the per-claim findings. A nil verifier uses recall.DefaultArtifactVerifier (the same default the read gate uses). When the note names no checkable artifact the input is returned unstamped and findings is empty — there was nothing to prove.
func Register ¶
func Register(d Driver)
Register adds (or replaces) a driver. A plugin/agent registers its own strategy here the same way the built-ins do — the registry is the open extension seam.
func RenderNotesDigest ¶ added in v0.38.0
RenderNotesDigest renders MEMORY.md plus linked fact bodies through the same trust gate `fak memory recall` applies per query (#2429): every fact body is paged in via NotesBackend.Materialize, so a sealed or stale-claim note never enters the digest wearing the authority of a fact. This is the session-start counterpart to memoryread.RenderDigest (which reads raw bytes off disk with no screen and no claim probe), wired into `fak memory-read`. Byte-budget and overflow-naming behavior otherwise match RenderDigest exactly.
func SameSubject ¶ added in v0.37.0
func SameSubject(a, b PromotionRecord) bool
SameSubject reports whether two PromotionRecords describe the same durable subject — the precondition for a conflict to even be possible. Two records are the same subject when they share a CellID (the same cell re-promoted over its life) or share a non-empty SourceSpan.Descriptor (two cells asserting a fact about the same named thing). An empty descriptor never matches another empty descriptor — two records that both failed to record a descriptor are not thereby "the same subject," they are two unlabeled facts, and treating them as a match would manufacture false conflicts out of missing data.
func ValidConflictOutcome ¶ added in v0.37.0
func ValidConflictOutcome(o ConflictOutcome) bool
ValidConflictOutcome reports whether o is a member of the closed vocabulary.
Types ¶
type AdmissionVerdict ¶ added in v0.38.0
AdmissionVerdict is the deny-by-structure ruling on a candidate durable memory write. Reason is drawn from the closed vocabulary above and is AdmitOK exactly when Admit is true; Detail names the offending structure (the measured size, the matched phrase) so a refusal is auditable from the verdict alone.
func AdjudicateMemoryWrite ¶ added in v0.38.0
func AdjudicateMemoryWrite(body []byte) AdmissionVerdict
AdjudicateMemoryWrite is the deny-by-structure rule set (#2912). It judges a candidate durable memory write by structure alone and returns a verdict whose Reason is from the closed vocabulary above. It is pure and deterministic: the same body always yields the same verdict, with no model call, clock, or I/O — so a #3006-style verbatim write is refused the same way every time.
type Backend ¶
type Backend interface {
// Cells returns the page table as cells, carrying only SAFE metadata (never the
// bytes of a sealed span). It is a snapshot; the executor does not mutate it.
Cells(ctx context.Context) ([]Cell, error)
// Materialize pages one cell's bytes in THROUGH THE TRUST GATE. A sealed/refused
// cell returns an error wrapping ErrSealed; the bytes never cross the gate.
Materialize(ctx context.Context, id string) ([]byte, error)
}
Backend supplies cells and trust-gated byte access. The two required methods are reads; the optional Tombstoner/Pruner add the durable mutations a backend chooses to support. A backend that implements neither is read/propose-only — the safe floor.
type Caps ¶
Caps is the capability grant that lets Run APPLY a durable mutation. Without a grant for an effect kind, that effect is recorded as a PROPOSAL and the backend is not touched — the fail-closed default. Apply is the master switch; Allow names the specific mutation kinds permitted.
type Cell ¶
type Cell struct {
ID string `json:"id"` // stable address within the backend (e.g. "step:3")
Step int `json:"step"` // ordinal position (recall page step; -1 for unordered)
Role string `json:"role,omitempty"` // the producer (tool name, "user", "system", "memq/consolidate")
Kind string `json:"kind,omitempty"` // tool_result | reasoning | user | system | episode | disposition
Descriptor string `json:"descriptor"` // SAFE extractive/sealed descriptor; never sealed bytes
Digest string `json:"digest,omitempty"`
Bytes int64 `json:"bytes"` // size (token-cost proxy)
Durability string `json:"durability,omitempty"` // turn | session | bounded | durable
Sealed bool `json:"sealed,omitempty"` // quarantined by the trust gate
Tombstoned bool `json:"tombstoned,omitempty"` // suppressed by context control
Witness string `json:"witness,omitempty"` // external trust witness
Refs []string `json:"refs,omitempty"` // digests/ids this cell references
Attrs map[string]string `json:"attrs,omitempty"` // OPEN attribute bag (forward-compatible, drop-unknown)
}
Cell is one addressable unit of memory: a recall page, an in-memory note, or a derived disposition. It carries only SAFE metadata — never the bytes of a sealed span (Descriptor is the extractive-or-sealed-metadata descriptor, exactly as recall.Page records it). Bytes are fetched lazily through a Backend's trust-gated Materialize, never carried on the cell.
type CodexBackend ¶ added in v0.35.0
type CodexBackend struct {
// contains filtered or unexported fields
}
CodexBackend is a READ-ONLY memq Backend over an external Codex memories home — the generated memory artifacts the OpenAI Codex agent writes under CODEX_HOME (normally ~/.codex). It surfaces those files as recall CANDIDATES, never as authoritative team rules: every cell is stamped provenance=external/untrusted, every page-in is re-screened through fak's trust gate (ctxmmu.ScreenBytes — the same screen recall runs), and a secret/injection-shaped memory is SEALED so it can never render into context. fak never writes a Codex file: the backend opens no file for write, implements neither Tombstoner nor Pruner, and treats a missing/partial home as an empty (not an error) corpus — the fail-closed posture for untrusted external state.
Layout (authoritative per the Codex manual, https://developers.openai.com/codex/memories):
- <home>/memories/*.md → kind codex-memory
- <home>/memories_extensions/chronicle/*.md → kind codex-chronicle
Chronicle memories are generated from screen content and so carry a HIGHER prompt-injection risk; the backend tags them with a distinct source kind and a suspicion attr so a policy can exclude them wholesale, independent of the per-cell content screen.
func NewCodexBackend ¶ added in v0.35.0
func NewCodexBackend(home string, includeChronicle bool) (*CodexBackend, error)
NewCodexBackend builds a read-only backend by scanning a Codex home. A missing/partial home yields an EMPTY backend (no error) — untrusted external state must never crash the algebra. includeChronicle gates the screen-derived Chronicle tree; when false those files are not enumerated at all.
home is used verbatim (resolve flag > CODEX_HOME > the platform default at the CALL site, so this constructor never silently reaches into a real ~/.codex). An empty home is treated as "no home configured" and yields an empty backend.
func (*CodexBackend) Cells ¶ added in v0.35.0
func (b *CodexBackend) Cells(_ context.Context) ([]Cell, error)
Cells returns the scanned page table (safe metadata only) — a snapshot copy so the executor never mutates the backend's slice.
func (*CodexBackend) Home ¶ added in v0.35.0
func (b *CodexBackend) Home() string
Home reports the resolved Codex home this backend scanned ("" when none was configured). Read-only accessor for the CLI label.
func (*CodexBackend) Materialize ¶ added in v0.35.0
Materialize pages one Codex file in THROUGH THE TRUST GATE. A cell sealed at scan time is refused with ErrSealed; otherwise the raw bytes are RE-SCREENED at read time (the same independent content re-screen recall runs, so a file that changed shape since the scan, or that the scan-time heuristic missed, is still caught) before any byte crosses the gate. The Codex file itself is never modified.
type ConflictDecision ¶ added in v0.37.0
type ConflictDecision struct {
Outcome ConflictOutcome `json:"outcome"`
Reason string `json:"reason"`
// Winner is the record DetectFactConflict identifies as the one to keep acting on
// when Outcome is ConflictPreferNewer (the zero value otherwise — ConflictAskUser
// and ConflictKeepBothScoped never pick a single winner by construction).
Winner PromotionRecord `json:"winner,omitempty"`
// A and B are the two records the decision was computed over, always in the same
// order they were passed to DetectFactConflict — never reordered, so a caller can
// tell which side of the call produced Winner.
A PromotionRecord `json:"a"`
B PromotionRecord `json:"b"`
}
ConflictDecision is the typed verdict DetectFactConflict produces: the outcome, an operator-readable reason (mirroring recall.StaleFactDecision.Reason), and the two records it was computed over so a caller/audit surface never has to re-derive which promotion "won" from prose.
func DetectFactConflict ¶ added in v0.37.0
func DetectFactConflict(a, b PromotionRecord) ConflictDecision
DetectFactConflict is the PURE detection function (#1597): given two PromotionRecords already identified as SameSubject, it returns EXACTLY ONE closed-vocabulary ConflictDecision. No clock read, no I/O, no hidden state — the same (a, b) always reproduces the same decision, and swapping the argument order never changes the resolved Outcome or Winner (only which of A/B the caller sees the inputs echoed back as), so a caller cannot get a different answer by calling it "the other way around."
Decision order (first match wins, most conservative first — mirrors recall.DetectStaleFact's structure):
- Not SameSubject -> ConflictNone. Nothing to resolve; the two records are about different things.
- SameSubject but identical SourceSpan.Digest (or both empty) -> ConflictNone. A re-affirmation of the same content is agreement, not disagreement.
- SameSubject, differing digest, BOTH sides carry ConsentExplicit -> compare source spans: if the spans differ in Step AND Role (distinct scopes: different turns produced by different roles/contexts), the disagreement is plausibly two scoped facts rather than one fact with two contradictory values -> ConflictKeepBothScoped. Otherwise (both explicit, same/overlapping scope, genuinely contradictory) -> ConflictPreferNewer, with Winner set to the higher-Seq record (later promotion wins because both sides already cleared the strongest consent bar, so recency is a safe, explicit tiebreaker — never a silent one, since the decision and its reason travel with the resolved value).
- Either side is ConsentUnknown, or neither side is ConsentExplicit (both merely ConsentInferred with no explicit affirmation to break the tie) -> ConflictAskUser. The records disagree and nothing in their own metadata is strong enough to settle it without a human — the same "ask rather than assume" posture recall.StaleFactExpiredMustQuery already takes.
func DetectFactConflicts ¶ added in v0.37.0
func DetectFactConflicts(recs []PromotionRecord) []ConflictDecision
DetectFactConflicts scans every SameSubject pair within a slice of PromotionRecords (e.g. PromotionLedger.For(cellID), or a caller-assembled cross-cell candidate set for a shared descriptor) and returns a ConflictDecision for each pair whose Outcome is not ConflictNone — the caller-facing entry point that spares every site from re-deriving pairwise iteration, mirroring the "detect over the whole set, act only on the actionable subset" shape this package's other multi-record scans already use. Pairs are compared in slice order (i<j); a record is never compared against itself.
type ConflictOutcome ¶ added in v0.37.0
type ConflictOutcome string
ConflictOutcome is the CLOSED vocabulary of resolutions for two durable facts that disagree — mirroring recall.StaleFactOutcome's shape (closed string type, membership set, fail-closed String()). Exactly one outcome is produced per detection; there is no "no decision" state once DetectFactConflict runs on an actual conflict.
const ( // ConflictNone: the two records are not in conflict — either they do not share a // subject, or they share a subject but carry identical content (same digest, a // re-affirmation rather than a disagreement). Nothing to resolve. ConflictNone ConflictOutcome = "none" // ConflictPreferNewer: both records carry ConsentExplicit and the disagreement is // within the same scope — safe to let the later promotion supersede the earlier // one, but EXPLICITLY (the decision travels with a reason and names the winner), // never a silent overwrite. ConflictPreferNewer ConflictOutcome = "prefer_newer" // ConflictAskUser: the disagreement cannot be resolved from the records' own // metadata alone — e.g. either side's consent is ConsentUnknown, or both sides are // merely ConsentInferred with no explicit affirmation to break the tie. The safe // move is to ask before silently picking a winner, mirroring // recall.StaleFactExpiredMustQuery / ctxplan.PageFaultQueryUser's "ask rather than // assume" posture. A caller wiring an interactive surface may route this outcome // through selfquery's clarification broker (selfquery.ClarificationQuestion), // exactly as selfquery.PromotionClarification (#1596) already routes an ambiguous // CONSENT verdict through that broker — this package does not import selfquery // itself (selfquery is the composer that imports memq, not the reverse), it only // mints the same shape of typed, ask-worthy decision selfquery already knows how to // wrap. ConflictAskUser ConflictOutcome = "ask_user" // ConflictKeepBothScoped: both records carry independent explicit consent (the // user affirmatively asserted each one) and their source spans differ enough // (different Step AND Role) that they plausibly describe two SCOPES of the same // subject rather than one subject with two contradictory values (e.g. "prefers // tea" from a home-context turn and "prefers coffee" from a work-context turn). // Both are retained; neither is discarded; the decision records that they are // scoped rather than merged. ConflictKeepBothScoped ConflictOutcome = "keep_both_scoped" )
func (ConflictOutcome) String ¶ added in v0.37.0
func (o ConflictOutcome) String() string
type Driver ¶
type Driver struct {
Name string `json:"name"`
Doc string `json:"doc"`
Build func(Params) Query `json:"-"`
}
Driver is a NAMED, pre-composed memory strategy — a "canned query" in the algebra. The whole point of memq is that a Driver is a few lines of Ops, not a bespoke Go function: recall, render, clean, compact, and dream below are five sentences in one language, and an agent (or a plugin) authors a sixth by emitting its own Query — no kernel edit. Build is pure and deterministic.
type Effect ¶
type Effect struct {
Kind string `json:"kind"`
Applied bool `json:"applied"`
Reason string `json:"reason,omitempty"`
Cells []string `json:"cells,omitempty"`
Derived *Cell `json:"derived,omitempty"`
DerivedBytes []byte `json:"-"`
Note string `json:"note,omitempty"`
}
Effect is one mutation/derivation the pipeline performed or proposed. Applied is true only when a Caps grant let Run mutate durable backend state. Derived carries a consolidation's artifact (its bytes are in DerivedBytes, json-excluded).
type Explanation ¶ added in v0.37.0
type Explanation struct {
CellID string `json:"cell_id"`
Found bool `json:"found"`
SourceSpan SourceSpan `json:"source_span,omitempty"`
Durability string `json:"durability,omitempty"`
Consent string `json:"consent,omitempty"`
Producer string `json:"producer,omitempty"`
Expiry string `json:"expiry,omitempty"`
Reason string `json:"reason,omitempty"`
// Narrative is a fixed-template sentence assembled ONLY from the fields above —
// string concatenation, not a model summarization call. It exists so a CLI/agent
// has one human-readable line, without reopening the model-narration door the
// issue explicitly closes.
Narrative string `json:"narrative"`
}
Explanation is the structured, ledger-only account of why a cell is in durable memory — the done-condition artifact for #1595. Every field traces to a PromotionRecord field; Explain performs no model call and synthesizes no prose beyond formatting the record, so a caller can render it verbatim as the answer to "why do you remember this."
type IndexOverflow ¶ added in v0.38.0
type IndexOverflow struct {
Reason string `json:"reason"` // always OverflowReason
Budget int64 `json:"budget"` // the byte cap enforced
Kept int `json:"kept"` // in-budget prefix count
Dropped []OverflowEntry `json:"dropped"`
}
IndexOverflow is the typed verdict for an over-budget memory index (#2430): the named entries that did not fit under the byte cap, plus the cap and the in-budget count. It is emitted ONLY when at least one entry overflows — a zero-overflow index yields no verdict (Result.Overflow stays nil), so the common in-budget path carries no advisory spam. The Dropped list is deterministic (it preserves the working-set order Run saw) and IS the compaction work-list, mirroring how Refused/Advisory surface named cells rather than counts.
type MemStore ¶
type MemStore struct {
// contains filtered or unexported fields
}
MemStore is the in-memory reference Backend: a page table plus a content-addressed blob map. It implements Backend, Tombstoner, and Pruner, so the whole algebra runs with zero setup (no disk, no recall image) — the substrate for the demo and the tests. A sealed cell's bytes stay in the CAS (audit), but Materialize refuses them, exactly as recall does. It also owns a PromotionLedger (#1595): every Add whose resulting durability crosses past turn-class mints a PromotionRecord, so a caller can later explain that write from structure alone.
func NewDemoStore ¶
func NewDemoStore() *MemStore
NewDemoStore builds a deterministic in-memory corpus that exercises the whole algebra without a disk image or a model — the default backend for `fak memory` and the memqdemo. It mixes the two axes from CONTEXT-IS-NOT-MEMORY.md: durable standing preferences, ephemeral turn-class observations ("it's 3pm"), session-class task state, benign tool results (one duplicated, so it carries a refcount), a sealed poison span, and an orphan CAS blob for the prune op to reclaim.
It is intentionally about the "refund-fee" support task so the intent string "refund fee" ranks a clear subset, mirroring the recall/cdb demo corpora.
func NewMemStore ¶
func NewMemStore() *MemStore
NewMemStore returns an empty store with an empty promotion ledger.
func (*MemStore) Add ¶
Add appends a cell whose bytes are `body`, computing the digest and a safe descriptor. A sealed cell gets a sealed-metadata descriptor (never its bytes), just as recall.Recorder does. id is assigned as "cell:<n>" by insertion order. This is the ConsentInferred, producer-defaulted convenience used by the demo corpus and existing callers; a caller that has real consent/producer/expiry provenance should call AddPromoted instead so the ledger records it faithfully.
func (*MemStore) AddIfDurable ¶ added in v0.37.0
func (m *MemStore) AddIfDurable(role, kind, durability string, body []byte, sealed bool, reclass string) (Cell, error)
AddIfDurable is Add's ephemeral-gated counterpart: it runs GateEphemeralPromotion over body's text BEFORE calling Add, and refuses (ErrEphemeralRefused, zero Cell) a durability-targeting write whose body is a situational observation with no explicit reclassification. A caller that already knows its durability target is turn/session-class (pure context, never eligible for promotion per CONTEXT-IS-NOT-MEMORY.md) should keep calling Add/AddPromoted directly — this gate only needs to run on the path that targets DurabilityBounded/DurabilityDurable, mirroring PromotionLedger.Record's own "only non-turn crosses the boundary" posture.
func (*MemStore) AddOrphanBlob ¶
AddOrphanBlob inserts a CAS blob that NO cell references — an unreferenced blob the prune op reclaims. Returns its digest.
func (*MemStore) AddPromoted ¶ added in v0.37.0
func (m *MemStore) AddPromoted(role, kind, durability string, body []byte, sealed bool, meta PromotionMeta) Cell
AddPromoted is Add plus an explicit PromotionMeta: it appends the cell exactly as Add does, then — if the resulting durability is not DurabilityTurn — records a PromotionRecord on the store's ledger capturing the source span, durability class, consent, producer, and expiry (#1595). A turn-class cell records nothing (it was never promoted; see PromotionLedger.Record).
func (*MemStore) AddPromotedIfDurable ¶ added in v0.37.0
func (m *MemStore) AddPromotedIfDurable(role, kind, durability string, body []byte, sealed bool, meta PromotionMeta, reclass string) (Cell, error)
AddPromotedIfDurable is AddPromoted's ephemeral-gated counterpart (#1592): for a write whose NORMALIZED durability target is DurabilityBounded or DurabilityDurable, it runs the write-time ephemeral gate over body's text first. A situational observation (timestamp/current-step/mood-bound, or any other non-durable-shaped text) is refused with ErrEphemeralRefused UNLESS reclass names an explicit reclassification (ReclassifyExplicitConsent/ReclassifyUserConfirmed/ ReclassifyEstablishedPattern) — "unless explicitly reclassified" per the issue's done condition. A turn/session-targeted write is not gated at all: it was never eligible for promotion in the first place (CONTEXT-IS-NOT-MEMORY.md's own decision tree), so gating it would refuse ordinary context writes that were never headed for durable memory.
func (*MemStore) ApplyStarveUpdates ¶ added in v0.38.0
func (m *MemStore) ApplyStarveUpdates(updates []StarveUpdate) int
ApplyStarveUpdates persists proposed counter updates onto the page table's Attrs — the write-back half of the credit, kept on the store (not the executor) so Run stays a pure read. Unknown IDs are skipped; it returns how many cells were updated. The Attrs map is cloned per write (snapshots handed out by Cells share maps with the table, and a past Result must not observe a later pass's counter).
func (*MemStore) Materialize ¶
Materialize pages a cell's bytes in, refusing a sealed cell (the trust gate).
func (*MemStore) Promotions ¶ added in v0.37.0
func (m *MemStore) Promotions() *PromotionLedger
Promotions returns the store's promotion ledger — the audit trail behind every non-turn-class cell this store holds (#1595). Callers (notably `fak memory explain-promotion`) read it to explain a fact without asking a model to narrate.
func (*MemStore) Prune ¶
Prune reclaims CAS blobs no cell references. With apply=false it only counts.
func (*MemStore) Reclassify ¶ added in v0.38.0
Reclassify persists a LOWERED durability class for a cell and mints a demotion audit record on the store's promotion ledger (#4147 — the caps-gated reclassify write-back). It is demote-only by construction: a newClass that is not STRICTLY lower than the cell's current class is declined (false, nil) — the same durabilityRank guard the executor enforces, re-checked here so the persisted store can never be PROMOTED through this seam. The change is reversible: the ledger keeps the whole reclassification history (For/Latest never overwrite), and the cell's prior promotion record still stands. A demotion whose target normalizes to turn-class is applied to the cell but not ledger-audited (turn is pure context, never memory — RecordReclassify mints nothing for it). Returns false for an unknown cell.
type NearDupPair ¶ added in v0.38.0
NearDupPair is one advisory near-duplicate pair (#2506): two cells with distinct content addresses whose bodies are similar above the opt-in threshold (simhash cosine). It is ADVISORY only — surfaced for a compactor (memory-compact) to consume — never a collapse (a fuzzy signal never silently decides; the vdso/neardup.go discipline). A is the smaller ID, B the larger, so a pair is canonically ordered.
type NotesBackend ¶ added in v0.37.0
type NotesBackend struct {
// contains filtered or unexported fields
}
NotesBackend is a READ-ONLY memq Backend over a markdown memory store — the MEMORY.md-indexed fact-file grammar internal/memoryread renders (the committed fleet mirror `.claude/memory`, or a harness auto-memory store passed by path). It is the loop-facing recall corpus (#2347, epic #2346 R1): the SAME algebra that runs over recall core images and Codex memories runs, unchanged, over the store the agent loop actually reads.
Two gates run on every page-in, and neither trusts the note's prose:
- the content screen (ctxmmu.ScreenBytes, the screen recall and the Codex backend run) — a secret/injection-shaped note is SEALED, never rendered;
- read-time artifact re-verification (recall.ExtractArtifactClaims + an injectable recall.ArtifactVerifier, default DefaultArtifactVerifier): a note naming a commit SHA, repo path, or flag that no longer verifies against the current checkout is refused ErrStale — a frozen self-report from a past session must not re-enter context wearing the authority of a fact (#2077).
The index IS the curation: only fact files linked from MEMORY.md become cells (an unindexed file is invisible to recall, exactly as it is to the harness). The backend opens no file for write and implements neither Tombstoner nor Pruner; a missing/partial store yields an empty corpus, never an error.
func NewNotesBackend ¶ added in v0.37.0
func NewNotesBackend(dir string) (*NotesBackend, error)
NewNotesBackend scans a memory store directory (MEMORY.md + linked fact files). A missing/empty store yields an EMPTY backend (no error) — a fresh node or a scrubbed clone must not crash the algebra. The verifier defaults to recall.DefaultArtifactVerifier; tests inject their own via WithVerifier.
func (*NotesBackend) Cells ¶ added in v0.37.0
func (b *NotesBackend) Cells(_ context.Context) ([]Cell, error)
Cells returns the scanned page table (safe metadata only) as a snapshot copy.
func (*NotesBackend) Dir ¶ added in v0.37.0
func (b *NotesBackend) Dir() string
Dir reports the store directory this backend scanned (for the CLI label).
func (*NotesBackend) HedgeReason ¶ added in v0.38.0
HedgeReason reports why a note must render hedged (reduced trust), or "" when it renders plainly. A note stamped unverified-at-write at the door hedges regardless of a later live re-check — the recorded doubt survives even when the live probe is inconclusive. Otherwise a note with no checkable claim hedges (nothing was ever proved), and a note whose live claims no longer all verify hedges with the offending claim named. A verified-at-write note whose claims still verify renders plainly.
The write-time stamp is read back from the note's source file (its frontmatter is stripped off the in-memory cell at scan) so this gate composes with the read-only backend without re-scanning the store.
func (*NotesBackend) Materialize ¶ added in v0.37.0
Materialize pages one note in through BOTH gates: the content re-screen (a note that turned secret/injection-shaped since the scan is sealed, not rendered) and the read-time artifact re-verification (a stale concrete claim refuses the whole note with the failing claim named). Fresh and unverifiable claims pass — the verifier is not an oracle over prose; tagging the hedge is the render surface's job via Verify.
func (*NotesBackend) Verify ¶ added in v0.37.0
func (b *NotesBackend) Verify(ctx context.Context, id string) ([]recall.ArtifactFinding, error)
Verify re-runs the read-time artifact verification for one note and returns the per-claim findings — the seam a render surface uses to tag a rendered note fresh vs unverifiable (a note with no concrete claims returns an empty slice: nothing checkable, render hedged).
func (*NotesBackend) WithVerifier ¶ added in v0.37.0
func (b *NotesBackend) WithVerifier(v recall.ArtifactVerifier) *NotesBackend
WithVerifier overrides the read-time artifact verifier (nil restores the default) — the same injectable seam recall.Session exposes.
type Op ¶
type Op struct {
Kind string `json:"kind"`
Pred *Pred `json:"pred,omitempty"` // filter
By string `json:"by,omitempty"` // rank key, or reclassify target class
Desc bool `json:"desc,omitempty"` // rank direction (default ascending)
K int `json:"k,omitempty"` // limit
Bytes int64 `json:"bytes,omitempty"` // budget cap, or narrow's per-cell width cap (bytes)
Window int `json:"window,omitempty"` // rank relevance_span: pooling width in cells (odd, >= 1; 1 = identity kernel)
Reason string `json:"reason,omitempty"` // tombstone/effect reason
// StarveK is the OPT-IN deterministic anti-starvation credit for the cutline ops
// (limit/budget), #4021. When > 0, every pass a cell falls below the op's cutline
// advances a per-cell counter in Attrs[StarveAttr]; once the cell JUST below the
// cutline is non-durable, unreferenced, and has sat below it for StarveK
// consecutive passes, it is retained for ONE pass (counter reset) — the
// NACL/Scissorhands anti-starvation axis with no RNG (see starve.go). 0 (the
// default) is byte-identical to today.
StarveK int `json:"starve_k,omitempty"`
// Protect opts a budget op into the protected-floor split (#4017, NACL/
// Scissorhands): every durable-class cell, plus the Recent most-recent cells
// by Step, is charged against the cap FIRST — surviving regardless of
// relevance rank — before the ephemeral remainder spends the leftover
// headroom in working-set order. False (the default) is today's plain
// prefix budget, byte-for-byte.
Protect bool `json:"protect,omitempty"` // budget: opt-in protected floor (#4017)
// Recent is the byte-exact recent-window size joined to the protected floor:
// the top-N most-recent cells by Step, regardless of class (the NACL
// pins ∪ recent union). Meaningful only with Protect; 0 protects the durable
// class alone.
Recent int `json:"recent,omitempty"` // budget: recent-window size (requires protect)
// MergeFloor opts a budget op into merge-on-evict (#4015, LOOK-M/CAM/KVMerger):
// instead of tail-dropping a below-budget cell, fold it into its single
// most-similar SURVIVING cell (argmax simhash cosine) when that similarity is at
// or above this floor. The evicted cell's Refs — and its own ID — ride onto the
// survivor, so its provenance is preserved rather than lost, and it leaves the
// overflow list. Cosine is in [-1,1]; a meaningful floor is in (0,1]. 0 (the
// default) is off: the budget tail-drops exactly as before, byte-for-byte.
MergeFloor float64 `json:"merge_floor,omitempty"` // budget: opt-in merge-on-evict similarity floor (#4015)
}
Op is one stage of the pipeline. The Kind discriminates which fields apply; the rest are an additive union (the same discriminated-union shape abi.Verdict uses).
type OverflowEntry ¶ added in v0.38.0
type OverflowEntry struct {
ID string `json:"id"`
Descriptor string `json:"descriptor,omitempty"`
Bytes int64 `json:"bytes"`
}
OverflowEntry names one memory entry that fell past the byte budget — never an anonymous tail-drop. Descriptor/Bytes let a compactor prioritize; ID is the recovery handle (the fact file / cell to consolidate or read directly).
type Params ¶
type Params struct {
Intent string `json:"intent,omitempty"`
K int `json:"k,omitempty"`
Budget int64 `json:"budget,omitempty"`
Reason string `json:"reason,omitempty"`
// NarrowBytes is the OPT-IN per-cell width cap for the second compression axis
// (#4019): when > 0, render and compact chain an OpNarrow immediately before
// their OpBudget, so an over-wide cell is slimmed (least-relevant Attrs fields
// dropped, Descriptor re-headLined) instead of tail-dropped whole. Default 0
// (off): the compiled Query is identical to before this field existed.
NarrowBytes int64 `json:"narrow_bytes,omitempty"`
// RetentionCount is the OPT-IN divergence-outlier exemption (MiniCache; #4018):
// when > 0, the compact driver carves the RetentionCount most-divergent fold
// candidates out of the set (OpExempt) before consolidating, so the cells a lossy
// fold would approximate worst survive bit-exact. 0 (the zero value) builds the
// exact pre-#4018 pipeline.
RetentionCount int `json:"retention_count,omitempty"`
// StarveK opts a driver's cutline op into the deterministic anti-starvation credit
// (#4021; see Op.StarveK). 0 (the default) builds the exact same query as before.
StarveK int `json:"starve_k,omitempty"`
// ProtectFloor opts the render driver's budget stage into the protected-floor
// split (#4017, NACL/Scissorhands): durable cells plus the ProtectRecent
// most-recent cells are charged against the byte budget FIRST, so a standing
// preference or the fresh tail is never tail-dropped by a low relevance rank.
// Both zero values compile to exactly the query the driver built before the
// knob existed — default off, byte-identical.
ProtectFloor bool `json:"protect_floor,omitempty"`
// ProtectRecent is the byte-exact recent-window size joined to the floor
// (top-N by Step). Meaningful only with ProtectFloor. Note the render
// driver's filter precedes its budget, so the window spans the filtered
// candidate set (intent-matching ∪ durable); author a raw scan→rank→budget
// query to protect the unconditional recency tail.
ProtectRecent int `json:"protect_recent,omitempty"`
// MergeFloor opts the render driver's budget stage into merge-on-evict (#4015):
// a cell evicted by the byte budget is folded into its nearest surviving near-twin
// (simhash cosine >= this floor) instead of tail-dropped, preserving the evicted
// cell's refs on the survivor. Cosine is in [-1,1]; a meaningful floor is in (0,1].
// 0 (the default) tail-drops as before — byte-identical.
MergeFloor float64 `json:"merge_floor,omitempty"`
}
Params parameterize a driver's compiled query: the agent's intent (for relevance), an optional K (limit), an optional byte Budget, and a Reason stamped on suppressions.
type Plan ¶
type Plan struct {
Intent string `json:"intent,omitempty"`
Steps []PlanStep `json:"steps"`
// Effects lists the distinct effect kinds the pipeline would perform, and
// Mutations the subset that change durable state (and so are proposal-only
// without a Caps grant). An empty Mutations means the query is read-only.
Effects []string `json:"effects,omitempty"`
Mutations []string `json:"mutations,omitempty"`
Valid bool `json:"valid"`
Error string `json:"error,omitempty"`
}
Plan is the static explanation of a query — the "step through this before you run it" surface. It is produced WITHOUT a backend, so it commits to no reads or writes; it is a description of the pipeline the agent authored.
type PlanStep ¶
type PlanStep struct {
Index int `json:"index"`
Kind string `json:"kind"`
Detail string `json:"detail"`
Effect bool `json:"effect"`
Mutation bool `json:"mutation"`
RequiresCap string `json:"requires_cap,omitempty"`
}
PlanStep is one explained stage of a query: what the op does, whether it is an effect, whether it mutates durable state (and so needs a capability to APPLY), and the capability key that gates it.
type Pred ¶
type Pred struct {
Op string `json:"op"`
Field string `json:"field,omitempty"` // durability|role|kind|descriptor|digest|witness|bytes|step|sealed|tombstoned|refcount|attr:<k> (unused by hasref, which reads Value against Refs)
Value string `json:"value,omitempty"`
Args []Pred `json:"args,omitempty"` // sub-predicates for and/or/not
}
Pred is a serializable predicate expression — the WHERE clause an agent authors as JSON. A zero Pred (or Op == "" / PredTrue) matches every cell.
type PromotionLedger ¶ added in v0.37.0
type PromotionLedger struct {
// contains filtered or unexported fields
}
PromotionLedger is an append-only, deterministic record of every context->memory promotion. It is NOT a Backend — it is the audit trail a Backend (MemStore today) attaches records to as it promotes cells, and the thing `fak memory explain-promotion` reads back. Append-only mirrors memq's own no-hard-delete posture (doc.go): a promotion record is never edited or removed, only ever added.
func NewPromotionLedger ¶ added in v0.37.0
func NewPromotionLedger() *PromotionLedger
NewPromotionLedger returns an empty ledger.
func (*PromotionLedger) All ¶ added in v0.37.0
func (l *PromotionLedger) All() []PromotionRecord
All returns every record in insertion order (a snapshot copy — the caller cannot mutate the ledger through it).
func (*PromotionLedger) Conflicts ¶ added in v0.37.0
func (l *PromotionLedger) Conflicts(cellID string) []ConflictDecision
Conflicts scans this ledger's OWN history for cellID — PromotionLedger.For(cellID) — for internal disagreements (a cell that was re-promoted with contradictory content over its life). This is the direct "reuse PromotionLedger's existing storage/lookup" entry point the issue asks for: a caller never needs a second ledger or to re-assemble the record slice itself to ask "does this cell's own promotion history conflict with itself." Cross-cell conflicts (same descriptor, different CellID) are still reachable via DetectFactConflicts on a caller-assembled slice; this method covers the common single-cell case for free.
func (*PromotionLedger) Explain ¶ added in v0.37.0
func (l *PromotionLedger) Explain(cellID string) Explanation
Explain answers "why is this cell in durable memory" using ONLY the ledger's structured record for cellID — never a model narration. A cell with no promotion record (never promoted, or unknown) yields Found=false and a narrative that says so plainly; it does not guess or fall back to summarizing the cell body.
func (*PromotionLedger) For ¶ added in v0.37.0
func (l *PromotionLedger) For(cellID string) (recs []PromotionRecord, ok bool)
For returns every promotion record for a cell ID, oldest first (a cell may be re-promoted/reclassified over its life; the ledger keeps the whole history, it never overwrites). ok is false when the cell has no promotion record at all — either it was never promoted (turn-class) or the ID is unknown to this ledger.
func (*PromotionLedger) Latest ¶ added in v0.37.0
func (l *PromotionLedger) Latest(cellID string) (PromotionRecord, bool)
Latest returns the most recent promotion record for a cell ID (ok=false if none).
func (*PromotionLedger) Record ¶ added in v0.37.0
func (l *PromotionLedger) Record(rec PromotionRecord)
Record appends a promotion record for a cell that is NOT DurabilityTurn. A DurabilityTurn cell is context-only by construction (CONTEXT-IS-NOT-MEMORY.md) and never earns a record — Record is a no-op for it, so a caller can call it unconditionally on every Add without special-casing turn-class writes itself. Durability/Consent are normalized to the closed vocabularies before storage, so a stored record can never carry an unrecognized class.
func (*PromotionLedger) RecordReclassify ¶ added in v0.38.0
func (l *PromotionLedger) RecordReclassify(cellID string, span SourceSpan, from, to, by string) bool
RecordReclassify mints an audit record for a durability DEMOTION applied to an already-stored cell — the demote-only twin of the promotion path (#4147). The ledger is append-only, so a reclassification is a NEW record layered over the cell's prior promotion, never an edit of it (For/Latest keep the whole history, so the class change is auditable AND reversible). from/to are the pre/post classes; the record stores the (lower) `to` class plus a structured Reason naming the transition, so Explain can account for the change from structure alone. It returns whether a record was minted: a `to` class that normalizes to turn writes nothing (turn is pure context, never audited as memory — the same rule Record enforces), so a demote-to-turn is applied by the backend but not carried here. Consent is ConsentInferred (a reclassify is a producer/driver action, not a fresh user request); Producer is `by`.
func (*PromotionLedger) Timeline ¶ added in v0.37.0
func (l *PromotionLedger) Timeline(cellID string) memview.Timeline
Timeline renders the FULL provenance timeline for a cell: every PromotionRecord this ledger holds for cellID, translated to memview.ProvenanceEvent and ordered chronologically. A cell with no promotion history yields an empty (but still renderable — "no history found") memview.Timeline, mirroring Explain's Found=false posture rather than erroring.
type PromotionMeta ¶ added in v0.37.0
PromotionMeta is the caller-supplied provenance for a promoting Add: whether the promotion was explicitly consented to, who/what produced it, and (for a bounded class) its expiry. A zero PromotionMeta defaults to ConsentInferred / an empty-producer-defaults-to-"unknown" / no expiry — the honest floor when a caller has no better information, never a silent upgrade to explicit consent.
type PromotionRecord ¶ added in v0.37.0
type PromotionRecord struct {
// CellID is the address of the promoted cell (memq.Cell.ID) — the join key back to
// the page table.
CellID string `json:"cell_id"`
// SourceSpan identifies WHERE the fact came from in the live turn: the ordinal
// step, the producing role, and a safe extractive descriptor (never sealed bytes —
// same posture as Cell.Descriptor). This is the "source span/reference" the issue
// asks for.
SourceSpan SourceSpan `json:"source_span"`
// Durability is the class earned at promotion time (turn|session|bounded|durable —
// reuses the existing memq/ctxmmu/recall vocabulary; NormDurability applied).
Durability string `json:"durability"`
// Consent records whether a human/operator explicitly asked for this promotion, or
// whether it was inferred by a producer/classifier (ConsentExplicit|ConsentInferred
// |ConsentUnknown; NormConsent applied).
Consent string `json:"consent"`
// Producer names WHAT wrote this cell: a tool name, "user", a driver
// ("memq/consolidate"), or a backend ("codex:<path>"). Free text but always
// populated — NormProducer defaults an empty value to "unknown" so the field is
// never silently blank.
Producer string `json:"producer"`
// Expiry is the optional validity bound for a `bounded` promotion (an opaque,
// caller-defined tick/step/timestamp string — memq does not interpret it, mirroring
// recall.Page.ValidTo's "auditable, re-checked elsewhere" posture). Empty means no
// stated expiry: a `durable` record is expected to leave this empty; a `bounded`
// record without one is a data-quality fact Explain surfaces, not an error.
Expiry string `json:"expiry,omitempty"`
// Reason is the free-text justification the producer supplied (e.g. a driver's
// tombstone/promotion reason, or "user said: remember this"). Never used as the
// ONLY explanation — Explain renders it alongside the structured fields, not
// instead of them.
Reason string `json:"reason,omitempty"`
// Seq is the ledger-assigned insertion order (0-based) — deterministic, so replay
// over the same inputs produces byte-identical records (no wall-clock dependence).
Seq int `json:"seq"`
}
PromotionRecord is the audit record CONTEXT-IS-NOT-MEMORY.md calls for: it captures WHY a fact crossed from live context into durable memq storage, so a caller can explain that promotion later from structure alone — never from asking a model to narrate/summarize the cell's history. Every field is either copied verbatim off the Cell at promotion time or supplied explicitly by the caller; nothing here is synthesized after the fact.
A "promotion" in memq's model is any MemStore.Add whose resulting cell durability is NOT DurabilityTurn — turn-class cells are pure context (they are never eligible for memory per CONTEXT-IS-NOT-MEMORY.md §4 Move 1) and so never mint a record. This mirrors the doc's decision tree: the record only exists on the branch that actually reaches "durable memory."
func (PromotionRecord) ProvenanceEvent ¶ added in v0.37.0
func (r PromotionRecord) ProvenanceEvent() memview.ProvenanceEvent
ProvenanceEvent translates one PromotionRecord into a memview.ProvenanceEvent for placement on a cell's provenance timeline. Seq is copied verbatim from the record (the ledger's own insertion order), so a timeline built purely from a cell's PromotionLedger.For(cellID) history is already in chronological order before any stale-fact transitions (internal/recall) are merged in.
type Pruner ¶
type Pruner interface {
Prune(ctx context.Context, apply bool) (blobs int, bytes int64, err error)
}
Pruner is the optional unreferenced-storage GC capability. Prune with apply=false is a dry-run count; apply=true reclaims. It never touches model-visible pages.
type Query ¶
type Query struct {
Intent string `json:"intent,omitempty"`
// NearDupThreshold is the OPT-IN advisory near-duplicate flag (#2506). When > 0,
// Run pages each non-sealed cell's body in through the trust gate, embeds it via
// internal/simhash, and reports body-similar pairs (cosine >= threshold) on
// Result.Advisory for a compactor (memory-compact) to consume. It is ADVISORY
// only — a fuzzy signal never silently decides (the vdso/neardup.go discipline),
// so the pair is reported, never collapsed. Default 0 (off).
NearDupThreshold float64 `json:"near_dup_threshold,omitempty"`
// Intents is the OPT-IN multi-intent recall slice (#4020, GQA group-reduce): the
// N consumer intents that must share this one retained set. When non-empty it
// supersedes Intent for relevance scoring — every cell is scored against every
// intent and the per-intent scores fold to ONE shared score via ScoreAgg (see
// reduceGroupScores in exec.go). Empty (the default) keeps the single-Intent
// scoring path byte-identical to today; Intents={x} ranks identically to Intent=x.
Intents []string `json:"intents,omitempty"`
// ScoreAgg is the tunable reduction operator folding per-intent scores when
// Intents is set: ScoreAggMean (the default when empty), ScoreAggAmax, or
// ScoreAggSum. An unknown value fails Validate (fail-closed); without Intents
// the field is inert.
ScoreAgg string `json:"score_agg,omitempty"`
Ops []Op `json:"ops"`
}
Query is an authored memory request: an intent string (for relevance ranking and as the default match terms) plus an ordered pipeline of Ops.
type RecallBackend ¶
type RecallBackend struct {
// contains filtered or unexported fields
}
RecallBackend adapts a loaded recall core image to the memq Backend interface, so the SAME algebra an agent authors over the in-memory store runs, unchanged, over the real durable store fak already ships. It is the proof that memq is not a toy: recall's own Recall/tombstone are re-expressible as memq queries against this backend.
It implements Backend (Cells + trust-gated Materialize) and Tombstoner (recall's negative-only RequestContextChange, persisted to dir). It deliberately does NOT implement Pruner: a recall image's CAS GC + reseal pass is recall.Dream's job, which memq complements — a prune op against this backend is reported as proposal-only with a pointer to `fak dream`.
func NewRecallBackend ¶
func NewRecallBackend(s *recall.Session, dir string) *RecallBackend
NewRecallBackend wraps a loaded session. dir is the image directory a tombstone is persisted back to; pass "" to keep tombstones in memory only (proposal/preview).
func (*RecallBackend) Cells ¶
func (r *RecallBackend) Cells(_ context.Context) ([]Cell, error)
Cells maps the recall page table to memq cells, carrying only safe metadata.
func (*RecallBackend) Materialize ¶
Materialize pages a recall page in through its trust gate, translating recall's seal errors into memq.ErrSealed so the executor labels the refusal without importing the recall error surface into memq core.
type Reclassifier ¶ added in v0.38.0
type Reclassifier interface {
Reclassify(ctx context.Context, id, newClass, by string) (bool, error)
}
Reclassifier is the optional durability-class write-back capability (#4147): the caps-gated apply for the reclassify op. Reclassify persists a LOWERED durability class for a cell and mints a demotion audit record, returning whether the demotion was actually applied (false if the backend declined — e.g. the cell is unknown, or newClass is not STRICTLY lower than the cell's current class). It is demote-only by construction: the same durabilityRank guard the executor enforces is re-checked on the backend side, so a persisted store can never be PROMOTED through this seam. A backend that does not implement it stays propose-only — the safe floor, exactly like Tombstoner/Pruner. The seam is shared with the consolidate durable write-back (#3906).
type Refusal ¶
type Refusal struct {
ID string `json:"id"`
Step int `json:"step"`
Role string `json:"role,omitempty"`
Reason string `json:"reason"`
}
Refusal is a cell an effect declined to touch (sealed/tombstoned/page-in refused).
type RenderItem ¶
type RenderItem struct {
ID string `json:"id"`
Step int `json:"step"`
Role string `json:"role,omitempty"`
Descriptor string `json:"descriptor"`
Bytes int64 `json:"bytes"`
Tokens int `json:"tokens"`
}
RenderItem is one cell materialized into context by an OpRender.
type Result ¶
type Result struct {
Intent string `json:"intent,omitempty"`
Steps []StepTrace `json:"steps"`
Rendered []RenderItem `json:"rendered,omitempty"`
Effects []Effect `json:"effects,omitempty"`
Refused []Refusal `json:"refused,omitempty"`
Advisory []NearDupPair `json:"advisory,omitempty"` // opt-in near-dup pairs (#2506); advisory, never collapsed
Overflow *IndexOverflow `json:"overflow,omitempty"` // typed over-budget verdict (#2430); nil when nothing overflowed
Starve *StarveReport `json:"starve,omitempty"` // typed anti-starvation verdict (#4021); nil unless an opt-in StarveK op changed a counter
Working []Cell `json:"working,omitempty"` // final working set (safe metadata only)
Stats Stats `json:"stats"`
}
Result is the outcome of Run.
func Run ¶
Run executes an authored query against a backend. It validates first (an invalid query never runs), loads the cells once, computes the global refcount + relevance scores, then threads a working set through the pipeline. Mutations are applied only where caps permit; otherwise they are proposed. The whole pass is deterministic: the same (backend cells, query, caps) yields a byte-identical Result.
type SourceSpan ¶ added in v0.37.0
type SourceSpan struct {
Step int `json:"step"`
Role string `json:"role,omitempty"`
Descriptor string `json:"descriptor,omitempty"`
Digest string `json:"digest,omitempty"`
}
SourceSpan is the safe, minimal pointer back to the live-context span a promotion came from — never the sealed bytes themselves.
type StarveReport ¶ added in v0.38.0
type StarveReport struct {
Reason string `json:"reason"` // always StarveReason
K int `json:"k"`
Granted string `json:"granted,omitempty"` // cell retained past the cutline this pass
Updates []StarveUpdate `json:"updates,omitempty"`
}
StarveReport is the typed anti-starvation verdict for one Run (#4021): the K threshold, every proposed counter update, and the ID of the cell (if any) the credit retained this pass. Emitted only when at least one counter changes — an idle pass leaves Result.Starve nil (no advisory spam), mirroring IndexOverflow. A second cutline op in the same pipeline appends its updates and overwrites K (last op wins), the same way Overflow keeps the last budget verdict.
type StarveUpdate ¶ added in v0.38.0
type StarveUpdate struct {
ID string `json:"id"`
Passes int `json:"passes"` // new consecutive-below-cutline count to persist
Granted bool `json:"granted,omitempty"` // the one-pass survival credit fired (counter reset)
}
StarveUpdate proposes one cell's new counter value. memq core never writes a backend (the same rung-1 honest scope as reclassify), so Run REPORTS the updates and the store applies them between passes (MemStore.ApplyStarveUpdates; a recall-image backend would persist Attrs the same way).
type Stats ¶
type Stats struct {
CellsScanned int `json:"cells_scanned"`
CellsSelected int `json:"cells_selected"` // final working-set size
Rendered int `json:"rendered"`
RenderedBytes int64 `json:"rendered_bytes"`
EstimatedTokens int `json:"estimated_tokens"`
Refused int `json:"refused"`
EffectsProposed int `json:"effects_proposed"`
EffectsApplied int `json:"effects_applied"`
// DedupCollapsed counts byte-identical cells folded by the read-side exact-digest
// collapse (#2506). It is the read-side dedup counter the dedup census fold reads
// alongside the write-side ledger (#2142).
DedupCollapsed int `json:"dedup_collapsed"`
// NearDupAdvisory counts body-similar pairs flagged by the opt-in simhash advisory
// (#2506) — the fuzzy near-dup counter for the dedup census fold.
NearDupAdvisory int `json:"near_dup_advisory"`
// Narrowed counts cells whose width was pruned by the opt-in OpNarrow pass (#4019)
// — the width-axis counter alongside the count-axis DedupCollapsed. omitempty keeps
// a serialized Result byte-identical when the op is never used.
Narrowed int `json:"narrowed,omitempty"`
// MergedOnEvict counts below-budget cells folded into a surviving near-twin by the
// opt-in merge-on-evict pass (#4015) instead of tail-dropped. omitempty keeps a
// serialized Result byte-identical when the op is never used.
MergedOnEvict int `json:"merged_on_evict,omitempty"`
}
Stats is the run accounting.
type StepTrace ¶
type StepTrace struct {
Index int `json:"index"`
Kind string `json:"kind"`
In int `json:"in"`
Out int `json:"out"`
Note string `json:"note,omitempty"`
}
StepTrace records one executed op: the working-set size in and out, plus a note.
type Tombstoner ¶
Tombstoner is the optional negative-only suppression capability (recall's RequestContextChange + persist). Tombstone reports whether the suppression was actually applied (false if the backend declined, e.g. the cell was already gone).