5 Rules for Agent Memory Systems: More Context Is Not the Fix
Filing, limits, and expiration beat a larger prompt window
An agent gave my team an answer with perfect confidence and the wrong deployment policy. The policy had changed weeks earlier. The old version was still present in a giant conversation transcript, while the new version was buried beside it.
We often diagnose this as a context-window problem. We ask for more tokens, larger summaries, or a better retriever. But the failure was simpler. The agent had no filing rule for changing facts.
A paper submitted on July 29, 2026 finally puts filesystem-based memory under a serious microscope. It models a management agent, a search agent, and an execution agent sharing one file store. It evaluates five benchmarks and six memory shapes, from chunk retrieval to verbatim files and agent-curated hierarchies. [1]
The result is useful precisely because it is not magical. A well-organized store reliably reduces retrieval cost, but it does not reliably improve answer accuracy. In large-memory settings, organized stores roughly halved retrieval cost. No evaluated agent consistently turned organization itself into better answers. [1]
That changes the engineering question. We should stop asking, “How do we give an agent more memory?” We should ask, “What records may it trust, how much may it read, and when must it forget?”
A longer prompt only delays failure
The common belief is understandable. If an agent misses a fact, add more history. If a summary loses detail, preserve the entire transcript. If retrieval fails, increase top_k.
Each move can help once. None creates a memory system.
A conversation is an event stream. A memory store is a set of claims with ownership, scope, and time. Those are different data structures. The first is optimized for replay. The second must support a decision made today.
The paper’s three-role model makes this distinction concrete. The management agent integrates incoming material. The search agent answers with cited sources. The execution agent produces task traces, which can later become reusable skills. Declarative facts and procedural skills therefore share one store, but they do not need identical records. [1]
This is a valuable correction for Go engineers building agent harnesses. We do not need a complex database on day one. We need clear write semantics before clever retrieval semantics.
Rule one — every record needs scope and time
Markdown is not durable memory by itself. A directory full of undated notes is simply a quieter form of chat history.
A useful record states what it covers, who owns it, and when it expires. I prefer small front matter because both humans and agents can inspect it before opening the full body.
---
kind: preference
subject: acme-production
valid_from: 2026-07-18
valid_until: 2026-10-18
supersedes: archive/deploy-policy-2026-05.md
confidence: confirmed
source: ticket/OPS-481
---
# Deployment policy
Production changes require a staged rollout and an approval record.The important line is not the prose. It is supersedes.
Without an explicit replacement edge, retrieval may surface two plausible policies. The model then performs conflict resolution in its hidden reasoning. This is not a reliable place to store operational truth. Expiry dates turn memory maintenance into a deterministic check.
I would separate records into current/, archive/, and raw/. The search agent should inspect current/ first. It should only consult archived or raw material when the question asks for history, evidence, or a contradiction review.
Rule two — organization controls cost, not truth
The paper compares chunk retrieval, a verbatim session dump, foldered sessions, reorganized stores, and agent-curated stores. The paper also includes a closed-book baseline. This is important because it prevents us from comparing a polished hierarchy only with one weak baseline. [1]
The stable result is surprisingly narrow. On the largest material streams, structured organization lowered retrieval expense by about half or more compared with a verbatim dump. The search agent often made more navigation calls, but each read pulled far less text. [1]
Study — five benchmarks and six memory forms
Workloads — LoCoMo, PersonaMem at 32k and 128k, REALTALK, and ALFWorld
Stable result — organization roughly halves retrieval cost when material is large
Unstable result — no memory shape wins answer accuracy in every workloadThis should change our KPI. A folder hierarchy is a budget mechanism, not a truth mechanism.
A structure can make retrieval cheap while preserving a bad summary, an obsolete preference, or a missing exception. Conversely, a lightly foldered store can beat a more aggressively rewritten store because it retains the original evidence.
So Crazy! The cheapest “organization” may be one that only places intact sessions under meaningful folders. It changes navigation while avoiding destructive rewriting.
Rule three — preserve evidence beside distilled notes
I do not trust a memory manager that may rewrite the only copy of a fact.
The paper reports a related problem. Unconstrained reorganization can compress away detail. In some long-conversation and real-chat workloads, preserving facts beat compressed reorganization. Yet a compact version performed better in another dense PersonaMem setting. [1]
This is not a contradiction. It is a signal that compression has a workload-dependent loss budget.
A practical store can preserve both layers:
memory/
raw/
2026-08-05-support-session.md
current/
customer/acme/deployment-policy.md
customer/acme/contact-preferences.md
skills/
incident-response/rollback-checklist.md
archive/
deploy-policy-2026-05.mdraw/ is append-only evidence. current/ contains concise, time-bounded operational claims. skills/ stores reusable procedures, not casual recollections. archive/ retains superseded records for audit and debugging.
Get Jacklyn Parrish’s stories in your inbox
Join Medium for free to get updates from this writer.
The distinction helps when an answer looks suspicious. We can trace a current claim back to raw evidence. We can also repair a bad distillation without pretending it never happened. That is much easier than debugging a vector embedding with no visible provenance.
Rule four — retrieval needs a hard budget
Flat chat-history retrieval has a seductive implementation. Read one large file and ask the model to find the answer. It works beautifully until it becomes expensive, slow, and vulnerable to stale context.
Below is a deliberately small Go comparison. It uses the standard library’s filesystem interface, so it is reproducible without adding an agent framework. os.ReadFile supplies the storage primitive documented by Go. [2]
package memory
import (
"os"
"strings"
)
func flatHistory(path string, maxBytes int) (string, error) {
b, err := os.ReadFile(path) // Reads the whole chat-history file.
if err != nil {
return "", err
}
if len(b) > maxBytes {
b = b[:maxBytes] // Budget cuts bytes, not relevance.
}
return string(b), nil
}
func structuredMemory(indexPath, query string, maxBytes int) (string, error) {
index, err := os.ReadFile(indexPath) // Small list of dated record paths.
if err != nil {
return "", err
}
for _, line := range strings.Split(string(index), "\n") {
parts := strings.SplitN(line, "|", 2)
if len(parts) != 2 || !strings.Contains(parts[1], query) {
continue // Metadata narrows the search before opening content.
}
b, err := os.ReadFile(parts[0])
if err != nil {
return "", err
}
if len(b) > maxBytes {
b = b[:maxBytes] // Each selected record has its own limit.
}
return string(b), nil
}
return "", os.ErrNotExist
}The flat function truncates whatever happens to be earliest in one file. The structured function reads a small index, chooses one record by metadata, and then applies a limit to the selected record.
The example is intentionally unsophisticated. A production system needs better matching, authorization, citations, and an explicit no-answer path. Still, it exposes the architectural difference. A retrieval budget only becomes meaningful after records become independently addressable.
My default policy is simple. Set a maximum number of candidate records, a maximum byte count per record, and a maximum total byte count. If no current record fits, return “insufficient evidence” and ask a follow-up question. Do not silently broaden the search into all history.
Rule five — tools shape the memory you get
We tend to treat the tool harness as plumbing. The study suggests otherwise. Changing the available tool set reshaped the resulting store as strongly as swapping the management model. [1]
This makes sense. A model with read, write, rename, and search behaves differently from one with a broad shell. Tools determine whether splitting, linking, searching, and repair are cheap actions or awkward ones.
For a Go service, I would expose narrow memory operations first:
CreateRecord(kind, subject, validity, source)
SupersedeRecord(oldPath, newPath)
SearchCurrent(query, budget)
ReadEvidence(path, lineRange)
ArchiveExpired(now)These are not merely convenience wrappers. They encode the store’s invariants. A generic shell remains valuable for maintenance, but it should not be the only interface for routine memory writes.
The write API is part of the memory policy. If an agent must provide valid_until before creating a mutable preference, stale state becomes harder to create by accident.
No file tree heals itself
Filesystem memory remains appealing. It is inspectable, portable, diffable, and friendly to tools developers already understand. Go’s os package provides a platform-independent interface for these basic filesystem operations. [2]
But we should not confuse inspectability with self-governance.
The growth study found organization eroded for every management agent except the strongest one. The paper also notes a limited time horizon. It studies streams at the scale of a conversation or a task sequence, not an agent maintaining years of personal history. [1]
This is the counterargument to my own prescription. A filing policy adds work. Very small, short-lived agents may not need one. A flat transcript plus a short context window can be cheaper and safer when there is no durable state to manage.
Use filesystem memory when facts recur across sessions, when users need to inspect state, or when an agent’s actions have operational consequences. Otherwise, do not build a miniature records office for a one-shot chatbot.
Make the policy testable
The paper does not publish a universal winning layout because one does not exist. Our job is to test the layout against our own workload.
I would add these checks to an agent-memory evaluation suite:
- Freshness — can the agent prefer a newer superseding record over an older one?
- Evidence — can every operational answer cite a current file and line range?
- Budget — how many bytes and files did retrieval consume?
- Loss — did distillation preserve every fact required by the task?
- Repair — can an expired or contradictory record be archived without deleting evidence?
The first useful benchmark is not a leaderboard. It is one real incident reconstructed twice: once from a flat transcript and once from structured files. Record answer correctness, files opened, bytes read, and whether the agent identified uncertainty.
Summary
- More context postpones disorder; it does not define memory.
- Structure reliably saves retrieval cost, not answer quality.
- Raw evidence and distilled records should coexist.
- Retrieval budgets need independently addressable, time-bounded files.
- Expiry, supersession, and narrow tools make memory policy enforceable.
The next generation of agents will not win by carrying an ever-longer prompt. They will win by knowing which record is current, how much evidence is enough, and when a memory has earned retirement.
References
- Filesystem-Based Memory for LLM Agents: Organization, Evolution, and Sustainability. https://arxiv.org/abs/2607.26637
- Go standard library —
ospackage. https://pkg.go.dev/os
