Building the Memory Layer of My Agent Harness: What I Learned Wiring Persistent Context by Hand
An LLM forgets everything the moment a request ends. Here is how I built a memory layer for my multi-agent harness in TypeScript, from a plain interface to semantic recall, token budgets and forgetting, and the mistakes I made along the way.
A few weeks into building my own agent harness, I asked my coding agent to “use the same folder structure as last time.” It answered politely and confidently, and it was completely wrong. There was no “last time” for it. Every request my harness sent to the model started from a blank slate.
That moment changed what I was building. I had a router and a small team of specialized agents working, inspired by tools like Ruflo (formerly claude-flow). But without memory, my “team” behaved like a group of brilliant contractors who had never met each other and had never seen the project before. Each one did good work in isolation, then forgot all of it.
I could have installed an existing memory library and moved on. Instead, I decided to wire persistent context by hand. My goal was not to build the best memory system in the world. My goal was to understand what a memory system actually does, piece by piece, so that the next time I use a framework I know what is happening underneath.
This post is the write-up of that journey. By the end you will understand:
- Why language models need an external memory at all, and what the context window really is
- The four kinds of memory an agent can have, and which ones are worth building first
- How to design a
MemoryStoreinterface and back it with SQLite - How embeddings and cosine similarity let an agent recall things by meaning, not by keywords
- How to rank memories by relevance, recency and importance
- How to pack the best memories into a prompt without blowing your token budget
- How lifecycle hooks write new memories back automatically
- How to share memory safely between agents, and how to make the system forget
I have written this for beginners. If you know basic TypeScript and have called an LLM API once, you can follow every step. Each code block is followed by an explanation of what it does and, more importantly, why it is designed that way.
Why an agent needs memory at all
An LLM is stateless: it remembers nothing between API calls, so any continuity you see is something the application rebuilt and sent back in.
This surprises a lot of people, because chat apps feel like they remember. They do not. When you chat with a model, the app quietly re-sends the whole conversation with every new message. The model reads it from the top each time, like an actor who is handed the full script before every single line.
The same two requests, with and without a memory layer. The model is identical in both columns; only the prompt changes.
The context window is RAM, not a hard disk
The best mental model I found is to treat the context window like a computer’s RAM. It is fast, the model can “see” everything in it at once, and it is the only place the model can think. But it is also limited and it is wiped after every request.
A memory layer is the hard disk. It stores everything durable outside the model, and before each request it loads only the small slice that matters into RAM. That is the entire idea. Everything else in this post is detail about doing that loading well.
This analogy explains three constraints that shaped my design:
- Space is limited. Even with very large context windows, every token costs money and latency. Worse, models get measurably less accurate when the important fact is buried in the middle of a huge prompt. More context is not automatically better context.
- Loading is selective. You cannot load the whole disk. You need a way to decide which memories are worth the space for this request.
- Writing is deliberate. Nothing is saved automatically. If you want the agent to remember something tomorrow, some code must decide to write it down today.
Why it is harder in a multi-agent harness
My harness has a router that sends each task to one of several specialized agents: a planner, a coder and a reviewer. With a single chatbot, “memory” mostly means chat history. With a team of agents, new questions appear:
- Should the reviewer see what the coder learned? (Usually yes.)
- Should the coder’s scratch notes leak into the planner’s prompt? (Usually no.)
- If two agents learn the same fact, do we store it twice?
- If a fact becomes wrong (we switched from npm to pnpm), who updates it?
These questions forced me to think about memory as a proper subsystem with its own interface, its own storage and its own rules, not as an array of messages I append to.
The mental model: four kinds of memory
Not all memory is the same, and treating it as one big bucket was my first mistake. Cognitive scientists split human memory into types, and that split maps surprisingly well onto agents.
Working and session memory live inside the prompt. Semantic and episodic memory live outside it and are recalled on demand.
1. Working memory is simply what is in the prompt right now: the system prompt, the tool definitions, the user’s task. It exists for exactly one API call. You already have this; every LLM app does.
2. Session memory is the recent history of the current conversation or task. In a chat app this is the message list. It is the easiest kind to implement (keep an array, send it back each time) and it is where most tutorials stop. Its problem is that it grows without limit and dies when the session ends.
3. Semantic memory holds facts and preferences that stay true across sessions. “This project uses pnpm.” “Vasanthan prefers TypeScript strict mode.” “The API base URL is set in config/env.ts." These are not tied to any particular moment. They are knowledge.
4. Episodic memory holds records of things that happened. “Last Tuesday the refactor of the router broke three tests because of a circular import.” Episodes carry lessons. They help an agent avoid repeating a mistake, which is something semantic facts alone cannot do.
Some writers add a fifth type, procedural memory, meaning learned skills or workflows (“how we deploy”). In my harness I store procedures as a special kind of semantic memory, so I will not treat it separately here.
Which ones should you build first?
Working and session memory you get almost for free. The interesting engineering is in semantic and episodic memory, because they live outside the model and must be recalled. That recall step is where all the hard questions live: what to fetch, how much, and in what order.
So the rest of this post is about building an external store for semantic and episodic memories and a pipeline that moves the right ones into the prompt at the right time.
The architecture: where memory sits in the harness
The single most useful decision I made was to put one component, the MemoryManager, between the agents and everything memory-related. Agents never touch the database, the embedding model or the hooks directly.
Each agent has its own namespace, but all memory traffic flows through one MemoryManager.
Here is what each piece does:
- Router. Reads the incoming task and picks the agent best suited to it. It existed before memory did.
- Agents (planner, coder, reviewer). Each one has a system prompt, a set of tools and a namespace such as
agent:coder. The namespace is like a folder label on every memory the agent writes. - MemoryManager. The public API of the memory layer. It exposes five verbs:
recall()finds candidate memories,rank()orders them,buildContext()packs the winners into a token budget,remember()writes new ones, anddecay()handles forgetting. - SQLite store. Durable storage on disk. One file, zero servers.
- Embedder. Turns text into a vector (a list of numbers) so we can search by meaning.
- Hooks. Functions the harness calls at fixed moments in the agent lifecycle, such as after every turn or at the end of a session. This is where automatic memory writing lives.
Why a manager instead of letting agents query the DB?
This is classic separation of concerns, and it paid off three times:
- Swappable storage. I started with an in-memory array, moved to SQLite, and could move to Postgres with
pgvectorlater. The agents never noticed. - One place for policy. Rules like “never store secrets” or “the reviewer may read the coder’s memories but not write to them” live in one file instead of being scattered across agents.
- Testability. I can unit test ranking and budgeting with fake data and no LLM calls at all.
If you remember one thing from this section: design memory as a service with a small interface, not as a pile of database calls. The next step shows that interface.
Step 1: Define what a memory is
Before writing any storage code, I wrote down the shape of a single memory. Getting this type right saved me from at least two painful migrations later.
// src/memory/types.ts
export type MemoryKind = "fact" | "preference" | "episode" | "procedure";
export interface MemoryRecord {
id: string; // unique id, e.g. a UUID
namespace: string; // "agent:coder", "shared", "user"
kind: MemoryKind; // what sort of memory this is
content: string; // the memory itself, in plain language
embedding: number[]; // vector form of `content`, used for semantic search
importance: number; // 0..1, how much this matters (set at write time)
createdAt: number; // epoch milliseconds
lastAccessedAt: number; // updated every time the memory is recalled
accessCount: number; // how many times it has been recalled
source?: string; // where it came from, e.g. "turn:8f2c" or "user"
}
export interface RecallQuery {
text: string; // what we are looking for, in natural language
namespaces: string[]; // which namespaces this agent may read
limit?: number; // max candidates to return (default 20)
kinds?: MemoryKind[]; // optional filter by kind
}
export interface ScoredMemory {
record: MemoryRecord;
relevance: number; // cosine similarity to the query
score: number; // final blended score after ranking
}
export interface MemoryStore {
insert(record: MemoryRecord): Promise<void>;
update(id: string, patch: Partial<MemoryRecord>): Promise<void>;
delete(id: string): Promise<void>;
listByNamespaces(namespaces: string[], kinds?: MemoryKind[]): Promise<MemoryRecord[]>;
}What each field is for
If you are new to this, every field can look arbitrary. Each one earns its place by answering a question the system will ask later.
namespaceanswers "who owns this and who may see it?" Without it, the planner's half-formed ideas would leak into the coder's prompt. We will use it in Step 7.kindanswers "how should this be treated?" A preference rarely expires; an episode about yesterday's failed build becomes less useful each week. Different kinds can get different decay rates.contentis stored in plain natural language, not JSON. This was deliberate. The content ends up inside a prompt, and models read sentences better than they read serialized objects. It also makes the database readable by a human when you are debugging at midnight.embeddingis the numeric fingerprint of the content. It lets us ask "which memories mean something similar to this question?" instead of "which memories contain these exact words?" Step 3 explains it fully.importanceanswers "how bad would it be to forget this?" "The production database is read-only" deserves 0.95. "The user said thanks" deserves 0.05.createdAt,lastAccessedAt,accessCountanswer "is this still alive?" A memory that keeps being recalled is clearly useful. One that nobody has touched in months is a candidate for forgetting.sourceanswers "why do we believe this?" When a memory turns out to be wrong, the source tells you which turn or which user message created it.
Why MemoryStore is so small
Notice that the MemoryStore interface has only four methods and none of them mention similarity search, scoring or tokens. That is on purpose. The store's only job is to persist records. All the intelligence (ranking, budgeting, deduplication) belongs in the MemoryManager above it.
This split is the Repository pattern. It means the store is easy to replace and the clever logic is easy to test. If you later move to a real vector database, you would add a search(embedding, k) method to the store for speed, but the manager's public API would not change.
Step 2: Make it persistent with SQLite
I chose SQLite because a memory layer for a personal harness does not need a server. It needs a single file that survives restarts, supports transactions, and can be inspected with any SQLite viewer. The better-sqlite3 package gives Node a fast, synchronous driver.
npm install better-sqlite3
npm install -D @types/better-sqlite3// src/memory/sqliteStore.ts
import Database from "better-sqlite3";
import { MemoryKind, MemoryRecord, MemoryStore } from "./types";
// Embeddings are arrays of floats. We store them as raw bytes (a BLOB)
// because that is ~4x smaller than JSON text and much faster to parse.
const toBlob = (v: number[]) => Buffer.from(new Float32Array(v).buffer);
const fromBlob = (b: Buffer) =>
Array.from(new Float32Array(b.buffer, b.byteOffset, b.byteLength / 4));
type Row = Omit<MemoryRecord, "embedding"> & { embedding: Buffer };
export class SqliteMemoryStore implements MemoryStore {
private db: Database.Database;
constructor(filePath = "./memory.db") {
this.db = new Database(filePath);
this.db.pragma("journal_mode = WAL"); // safe concurrent reads while writing
this.db.exec(`
CREATE TABLE IF NOT EXISTS memories (
id TEXT PRIMARY KEY,
namespace TEXT NOT NULL,
kind TEXT NOT NULL,
content TEXT NOT NULL,
embedding BLOB NOT NULL,
importance REAL NOT NULL,
createdAt INTEGER NOT NULL,
lastAccessedAt INTEGER NOT NULL,
accessCount INTEGER NOT NULL DEFAULT 0,
source TEXT
);
CREATE INDEX IF NOT EXISTS idx_mem_ns ON memories(namespace, kind);
`);
}
async insert(r: MemoryRecord): Promise<void> {
this.db
.prepare(
`INSERT INTO memories VALUES
(@id, @namespace, @kind, @content, @embedding, @importance,
@createdAt, @lastAccessedAt, @accessCount, @source)`
)
.run({ ...r, embedding: toBlob(r.embedding), source: r.source ?? null });
}
async update(id: string, patch: Partial<MemoryRecord>): Promise<void> {
const allowed = ["content", "importance", "lastAccessedAt", "accessCount", "embedding"];
const keys = Object.keys(patch).filter((k) => allowed.includes(k));
if (keys.length === 0) return;
const sets = keys.map((k) => `${k} = @${k}`).join(", ");
const params: Record<string, unknown> = { id, ...patch };
if (patch.embedding) params.embedding = toBlob(patch.embedding);
this.db.prepare(`UPDATE memories SET ${sets} WHERE id = @id`).run(params);
}
async delete(id: string): Promise<void> {
this.db.prepare(`DELETE FROM memories WHERE id = ?`).run(id);
}
async listByNamespaces(namespaces: string[], kinds?: MemoryKind[]): Promise<MemoryRecord[]> {
const nsMarks = namespaces.map(() => "?").join(",");
let sql = `SELECT * FROM memories WHERE namespace IN (${nsMarks})`;
const args: unknown[] = [...namespaces];
if (kinds?.length) {
sql += ` AND kind IN (${kinds.map(() => "?").join(",")})`;
args.push(...kinds);
}
const rows = this.db.prepare(sql).all(...args) as Row[];
return rows.map((row) => ({ ...row, embedding: fromBlob(row.embedding) }));
}
}Walking through the code
The table. Each column maps one-to-one onto a field of MemoryRecord. The only special case is embedding, stored as a BLOB. An embedding with 384 dimensions is 384 floats. As JSON text that is roughly 3 to 4 KB per memory; as a packed Float32Array it is exactly 1,536 bytes and needs no parsing.
toBlob and fromBlob. These two helpers convert between a normal JavaScript array and raw bytes. Float32Array stores each number in exactly 4 bytes, which is why fromBlob divides the byte length by 4. The byteOffset argument matters: Node may hand you a Buffer that is a slice of a larger memory pool, and forgetting the offset gives you garbage numbers. (I lost an evening to that one.)
WAL mode. journal_mode = WAL (write-ahead logging) lets readers keep reading while a write is happening. In a harness where an afterTurn hook is writing while the next agent is already recalling, this prevents "database is locked" errors.
The index. Every recall filters by namespace first, so an index on (namespace, kind) keeps that filter fast even with tens of thousands of rows.
Named parameters (@id, @content). Never build SQL by gluing user text into the string. Parameters let SQLite handle escaping, which protects you from SQL injection. Memory content often comes from LLM output, which you should treat as untrusted input.
The allowed list in update. It looks fussy, but it stops a caller from accidentally changing id or namespace, which would silently move a memory to another agent. Notice that column names are only taken from this fixed list, never from outside input, so the dynamic SET clause stays safe.
What about vector search?
You may have noticed that listByNamespaces returns every memory in the allowed namespaces, and we will compute similarity in JavaScript. For a personal harness with a few thousand memories, that is perfectly fine: comparing a query against 5,000 vectors of 384 numbers is about two million multiplications, which takes a few milliseconds.
When you outgrow that, you have options such as the sqlite-vec extension, Postgres with pgvector, or a dedicated vector database. The point of the MemoryStore interface is that you can make that switch later without touching anything above it. Start with brute force; optimize when you can measure the pain.
Step 3: Recall by meaning with embeddings
My first recall function used SQL LIKE '%keyword%'. It failed immediately. The agent asked "which installer should I run?" and the memory said "We use pnpm, not npm." Those two sentences share no important words, yet one is clearly the answer to the other. Keyword search cannot see that. Embeddings can.
What an embedding actually is
An embedding model reads a piece of text and outputs a fixed-length list of numbers, called a vector. A small, popular model like all-MiniLM-L6-v2 outputs 384 numbers for any sentence you give it.
The magic is in how the model was trained: sentences with similar meaning end up with vectors that point in similar directions. You can picture each vector as an arrow in space. “Package manager is pnpm” and “which installer should I run?” point almost the same way. “User prefers dark mode” points somewhere else entirely.
A 2-D sketch. Real embeddings have hundreds of dimensions, but the idea of “angle = difference in meaning” holds.
Measuring “pointing the same way”: cosine similarity
To compare two arrows we use cosine similarity, the cosine of the angle between them:
In plain words: multiply the two vectors number by number and add it all up (the dot product), then divide by both lengths so that only direction matters, not size. The result runs from -1 to 1:
- close to 1: same meaning
- around 0: unrelated
- negative: pointing in opposite directions (rare with sentence models)
The code
I run the embedding model locally with Transformers.js, so there are no API keys and no per-call cost. Swapping in a hosted embedding API only changes the Embedder class.
npm install @huggingface/transformers// src/memory/embedder.ts
import { pipeline, FeatureExtractionPipeline } from "@huggingface/transformers";
export interface Embedder {
embed(text: string): Promise<number[]>;
}
export class LocalEmbedder implements Embedder {
private extractor?: FeatureExtractionPipeline;
private async load() {
// Downloads the model once (~25 MB) and caches it on disk.
this.extractor ??= await pipeline("feature-extraction", "Xenova/all-MiniLM-L6-v2");
return this.extractor;
}
async embed(text: string): Promise<number[]> {
const extractor = await this.load();
const output = await extractor(text, { pooling: "mean", normalize: true });
return Array.from(output.data as Float32Array);
}
}
// src/memory/similarity.ts
export function cosineSimilarity(a: number[], b: number[]): number {
if (a.length !== b.length) throw new Error("Vector length mismatch");
let dot = 0, normA = 0, normB = 0;
for (let i = 0; i < a.length; i++) {
dot += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
if (normA === 0 || normB === 0) return 0;
return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}And the first version of recall inside the MemoryManager:
// src/memory/manager.ts (part 1)
async recall(q: RecallQuery): Promise<ScoredMemory[]> {
const queryVec = await this.embedder.embed(q.text);
const candidates = await this.store.listByNamespaces(q.namespaces, q.kinds);
return candidates
.map((record) => {
const relevance = cosineSimilarity(queryVec, record.embedding);
return { record, relevance, score: relevance };
})
.filter((m) => m.relevance >= this.minRelevance) // e.g. 0.30
.sort((a, b) => b.score - a.score)
.slice(0, q.limit ?? 20);
}Understanding the details
pooling: "mean". The model actually produces one vector per token (roughly per word piece). Mean pooling averages them into a single vector for the whole sentence. Without it you would get a grid of numbers instead of one arrow.
normalize: true. This scales every vector to length 1. Once all vectors have length 1, cosine similarity is just the dot product, because the denominators are both 1. I still divide in cosineSimilarity so that the function stays correct if I ever switch to a model that does not normalize.
this.extractor ??= .... Loading a model is slow, so we do it once and reuse it. The ??= operator means "assign only if currently undefined".
The minRelevance threshold. Every memory gets some similarity score, even nonsense ones. Without a floor, a query about installers would still return the dark-mode preference at the bottom of the list, and it would waste prompt space. I settled on 0.30 for this model after logging real scores for a week; you should tune yours the same way, because the right number depends on the embedding model.
One crucial rule. Every memory and every query must be embedded with the same model. Vectors from two different models live in different “spaces” and comparing them is meaningless. If you ever change models, you must re-embed every stored memory. I now store the model name in a small meta table so the harness can detect a mismatch at startup.
Step 4: Rank memories by relevance, recency and importance
Similarity alone turned out to be a bad ranking signal. In week two, my coder agent kept pulling up a three-week-old memory saying “tests are flaky, skip the integration suite” because it matched the word-level meaning of every testing question. The flakiness had long been fixed. The memory was relevant but stale.
The fix is to blend three signals, an idea popularized by the “Generative Agents” research from Stanford in 2023:
- Relevance is the cosine similarity from Step 3. Does this memory match the question?
- Recency is a number between 0 and 1 that shrinks as time passes since the memory was last used. Is this memory still fresh?
- Importance is the 0 to 1 value stored when the memory was written. How much does this memory matter in general?
- The weights (w) decide how much each signal counts. I use 0.6, 0.2 and 0.2, so relevance still dominates.
Left: how fast recency fades for different half-lives. Right: the “pnpm” fact wins even though the chit-chat memory is fresher, because relevance and importance outweigh recency.
Exponential decay, explained
For recency I use exponential decay with a half-life. The idea comes from physics: after one half-life the value is cut in half, after two it is a quarter, and so on.
Why exponential instead of a straight line? A straight line hits zero at some point and then goes negative, which makes no sense. Exponential decay never reaches zero; an old memory just gets quieter. It can still win if it is extremely relevant, which is exactly the behavior I wanted.
Get Vasanthan K’s stories in your inbox
Join Medium for free to get updates from this writer.
The half-life should depend on the kind of memory. An episode about today’s debugging session is useful for a day or two. A user preference stays true for months.
// src/memory/ranking.ts
import { MemoryKind, ScoredMemory } from "./types";
const HOUR = 60 * 60 * 1000;
export const HALF_LIFE_HOURS: Record<MemoryKind, number> = {
episode: 72, // 3 days
fact: 24 * 30, // ~1 month
procedure: 24 * 90, // ~3 months
preference: 24 * 180, // ~6 months
};
export interface RankWeights { relevance: number; recency: number; importance: number; }
export const DEFAULT_WEIGHTS: RankWeights = { relevance: 0.6, recency: 0.2, importance: 0.2 };
export function recencyScore(lastAccessedAt: number, kind: MemoryKind, now = Date.now()): number {
const hours = Math.max(0, (now - lastAccessedAt) / HOUR);
return Math.pow(0.5, hours / HALF_LIFE_HOURS[kind]);
}
export function rank(
memories: ScoredMemory[],
w: RankWeights = DEFAULT_WEIGHTS,
now = Date.now()
): ScoredMemory[] {
return memories
.map((m) => {
const recency = recencyScore(m.record.lastAccessedAt, m.record.kind, now);
const score =
w.relevance * m.relevance +
w.recency * recency +
w.importance * m.record.importance;
return { ...m, score };
})
.sort((a, b) => b.score - a.score);
}Why lastAccessedAt and not createdAt?
This is subtle and important. If recency were based on when a memory was created, a core fact like “we use pnpm” would fade forever, even though the agent relies on it every day. By measuring time since last use, a memory that keeps proving useful stays fresh. Memories that are never recalled fade naturally.
To make this work, the MemoryManager must "touch" each memory it actually puts in a prompt:
// src/memory/manager.ts (part 2)
async touch(used: ScoredMemory[]) {
const now = Date.now();
await Promise.all(
used.map((m) =>
this.store.update(m.record.id, {
lastAccessedAt: now,
accessCount: m.record.accessCount + 1,
})
)
);
}This mirrors something real about human memory: recalling a fact strengthens it. Software engineers know the same idea as an LRU (least recently used) cache.
Tuning the weights
There is no universally correct set of weights. My process was simple: log the top 10 ranked memories for 50 real tasks, mark which ones should have been included, and adjust. Two lessons came out of that:
- If irrelevant-but-recent memories keep appearing, lower the recency weight.
- If the agent keeps missing obvious long-standing rules, raise the importance of those memories at write time instead of fiddling with weights. Fix the data before you fix the formula.
Step 5: Build the context within a token budget
Ranking tells us which memories are best. It does not tell us how many to include. My first version included the top 20, always. Then one day an episode memory contained a 3,000-token stack trace, the prompt overflowed, and the request failed. Counting memories was the wrong unit. The unit is tokens.
Give memory a fixed budget
I now plan every prompt like a monthly salary: each part gets a fixed allowance, and memory has to live within its own.
Example numbers. Memory gets its own slice. Space for the model’s answer is reserved up front, because a prompt that fills the whole window leaves no room to reply.
Why a fixed slice instead of “whatever space is left over”? Because “left over” changes every turn. Predictable budgets make behavior predictable, and they make costs predictable too.
The packing algorithm
The approach is greedy packing: walk the ranked list from best to worst, and add each memory if it still fits. Skip any memory that does not fit, but keep going, because a smaller memory further down might still fit.
// src/memory/context.ts
import { ScoredMemory } from "./types";
// Fast approximation: ~4 characters per token for English text.
// Good enough for budgeting; use a real tokenizer if you need precision.
export const estimateTokens = (text: string) => Math.ceil(text.length / 4);
const MAX_TOKENS_PER_MEMORY = 400;
function formatMemory(m: ScoredMemory): string {
const date = new Date(m.record.createdAt).toISOString().slice(0, 10);
return `- [${m.record.kind}, ${date}] ${m.record.content}`;
}
export function buildMemoryContext(ranked: ScoredMemory[], budgetTokens: number) {
const header = "<memory>\nThings you know from earlier work. Prefer the user's current instructions if they conflict.\n";
const footer = "</memory>";
let used = estimateTokens(header + footer);
const lines: string[] = [];
const included: ScoredMemory[] = [];
for (const m of ranked) {
const line = formatMemory(m);
const cost = estimateTokens(line) + 1; // +1 for the newline
if (cost > MAX_TOKENS_PER_MEMORY) continue; // oversized: summarize it later (Step 8)
if (used + cost > budgetTokens) continue; // doesn't fit, but a smaller one might
lines.push(line);
included.push(m);
used += cost;
}
if (included.length === 0) return { text: "", included, tokens: 0 };
return { text: header + lines.join("\n") + "\n" + footer, included, tokens: used };
}Line by line
estimateTokens. Tokens are the chunks a model actually reads; for English, one token is roughly four characters. This estimate is off by 10 to 20 percent, which is fine for budgeting as long as you leave headroom. If you need exact counts, use the tokenizer that matches your model.
MAX_TOKENS_PER_MEMORY. This guard is the fix for my stack-trace disaster. No single memory is allowed to eat the budget. Oversized memories get skipped now and condensed later by the consolidation job.
continue instead of break. If memory number 3 is too big, break would stop and throw away memories 4 to 20, even if they are small and useful. continue skips just that one. This is a simple form of the classic knapsack problem: greedy is not perfectly optimal, but it is fast, predictable and good enough.
formatMemory. Each memory becomes one bullet with its kind and date. The date lets the model reason about staleness itself ("this note is from March; the user may have changed their mind"). The kind tells it whether it is reading a rule or a story.
The <memory> tags and the header sentence. Wrapping memories in clear tags helps the model tell recalled background apart from the user's live instructions. The sentence "prefer the user's current instructions if they conflict" settles a real problem: without it, an old memory could override something the user just said.
Returning included. The manager needs to know which memories were actually used so it can touch() them (Step 4). Only memories that make it into the prompt get their recency refreshed.
Where to put the block in the prompt
Research on long contexts has repeatedly found that models pay more attention to the beginning and the end of a prompt than to the middle, an effect often called “lost in the middle”. I place the memory block just before the user’s task, after the static system prompt and tool definitions. That keeps it close to the question it is meant to help answer.
Step 6: Write memories back with lifecycle hooks
Reading memory is only half the loop. Something has to decide what is worth remembering after each turn. I tried two wrong approaches before finding one that worked:
- Store every message. The database filled with “ok”, “thanks” and half-finished thoughts. Recall quality collapsed because noise outnumbered signal.
- Give the agent a
save_memorytool. Better, but the agent forgot to call it about half the time, and when it did call it, it interrupted the task it was doing.
The approach that stuck is a lifecycle hook: a function the harness calls automatically after every turn, separate from the agent’s main work.
The afterTurn pipeline. It runs after the user already has their answer, so it adds no visible latency.
A tiny hook system
Hooks are just named events with a list of listeners. Here is the whole implementation:
// src/harness/hooks.ts
export interface TurnEvent {
turnId: string;
agentId: string; // e.g. "coder"
userMessage: string;
agentReply: string;
toolCalls: { name: string; summary: string }[];
}
type HookMap = {
afterTurn: (e: TurnEvent) => Promise<void>;
onSessionEnd: (sessionId: string) => Promise<void>;
};
export class HookBus {
private listeners: { [K in keyof HookMap]: HookMap[K][] } = { afterTurn: [], onSessionEnd: [] };
on<K extends keyof HookMap>(name: K, fn: HookMap[K]) {
this.listeners[name].push(fn);
}
// Fire-and-forget: a failing hook must never crash the agent's turn.
emit<K extends keyof HookMap>(name: K, ...args: Parameters<HookMap[K]>) {
for (const fn of this.listeners[name]) {
(fn as (...a: unknown[]) => Promise<void>)(...args).catch((err) =>
console.error(`[hook:${name}] failed`, err)
);
}
}
}Why fire-and-forget? The user is waiting for the agent’s answer, not for the memory write. By not awaiting the hook, the reply goes out immediately and memory extraction happens in the background. The .catch is essential: an unhandled promise rejection in Node can crash the process, and a memory bug should never take down the agent.
Extracting durable facts with an LLM
The extraction step asks a small, cheap model to read the turn and pull out only things worth remembering. I treat the model as a function behind a minimal interface so any provider works.
// src/memory/extract.ts
import { MemoryKind } from "./types";
import { TurnEvent } from "../harness/hooks";
export interface LLM { complete(prompt: string): Promise<string>; }
export interface Candidate { kind: MemoryKind; content: string; importance: number; }
const EXTRACT_PROMPT = (e: TurnEvent) => `
You maintain long-term memory for a software agent.
Read this turn and list facts worth remembering in FUTURE sessions.
Rules:
- Only durable information: decisions, preferences, project conventions, lessons from failures.
- Skip greetings, thanks, and anything only relevant to this exact moment.
- Write each item as one self-contained sentence that makes sense with no other context.
- Never include passwords, API keys, tokens or personal data.
- importance: 0.9+ for hard rules, ~0.5 for useful context, <0.3 for trivia.
Return ONLY a JSON array like:
[{"kind":"fact|preference|episode|procedure","content":"...","importance":0.0}]
Return [] if nothing is worth keeping.
USER: ${e.userMessage}
AGENT: ${e.agentReply}
TOOLS: ${e.toolCalls.map((t) => `${t.name}: ${t.summary}`).join("; ")}
`;
export async function extractCandidates(llm: LLM, e: TurnEvent): Promise<Candidate[]> {
const raw = await llm.complete(EXTRACT_PROMPT(e));
try {
const json = raw.slice(raw.indexOf("["), raw.lastIndexOf("]") + 1);
const parsed = JSON.parse(json) as Candidate[];
return parsed.filter(
(c) => typeof c.content === "string" && c.content.length > 0 &&
typeof c.importance === "number"
);
} catch {
return []; // a malformed answer means "remember nothing", never a crash
}
}The most important rule in that prompt is “one self-contained sentence”. A memory like “use the second option” is useless three weeks later; nobody remembers what the options were. “Retry logic in fetchTool uses exponential backoff, max 3 attempts” makes sense on its own.
The JSON parsing is defensive on purpose. Models sometimes wrap JSON in prose or code fences, so I slice from the first [ to the last ]. If parsing still fails, the hook stores nothing. Losing one memory is far better than storing garbage.
Filter, dedupe and store
The last stage lives in the MemoryManager. It filters, checks for near-duplicates, and either inserts or reinforces.
// src/memory/manager.ts (part 3)
import { randomUUID } from "node:crypto";
const SECRET_PATTERNS = [
/sk-[A-Za-z0-9_-]{20,}/, // common API key shape
/AKIA[0-9A-Z]{16}/, // AWS access key id
/-----BEGIN [A-Z ]*PRIVATE KEY-----/,
/password\s*[:=]/i,
];
const MIN_IMPORTANCE = 0.3;
const DUPLICATE_THRESHOLD = 0.92;
async remember(c: Candidate, namespace: string, source: string): Promise<"stored" | "merged" | "skipped"> {
if (c.importance < MIN_IMPORTANCE) return "skipped";
if (SECRET_PATTERNS.some((p) => p.test(c.content))) return "skipped";
const embedding = await this.embedder.embed(c.content);
const existing = await this.store.listByNamespaces([namespace], [c.kind]);
let best: { id: string; sim: number; importance: number; accessCount: number } | null = null;
for (const r of existing) {
const sim = cosineSimilarity(embedding, r.embedding);
if (!best || sim > best.sim) best = { id: r.id, sim, importance: r.importance, accessCount: r.accessCount };
}
const now = Date.now();
if (best && best.sim >= DUPLICATE_THRESHOLD) {
// Same fact seen again: reinforce instead of duplicating.
await this.store.update(best.id, {
content: c.content, // newer wording wins, it may carry a correction
embedding,
importance: Math.max(best.importance, c.importance),
lastAccessedAt: now,
accessCount: best.accessCount + 1,
});
return "merged";
}
await this.store.insert({
id: randomUUID(), namespace, kind: c.kind, content: c.content, embedding,
importance: c.importance, createdAt: now, lastAccessedAt: now, accessCount: 0, source,
});
return "stored";
}And the wiring that connects it all:
// src/harness/setup.ts
hooks.on("afterTurn", async (e) => {
const candidates = await extractCandidates(cheapLLM, e);
for (const c of candidates) {
// Preferences and procedures are team-wide; facts and episodes stay with the agent.
const ns = c.kind === "preference" || c.kind === "procedure" ? "shared" : `agent:${e.agentId}`;
await memory.remember(c, ns, `turn:${e.turnId}`);
}
});Why each safeguard exists
The importance floor (0.3). The extractor sometimes keeps trivia despite instructions. A numeric floor is a cheap second line of defense.
The secret patterns. LLMs see API keys in tool outputs and config files all the time. A memory store that persists secrets turns one leaked log into a permanent leak in every future prompt. Regexes will not catch everything, but they catch the most common shapes. Defense in depth: the prompt says “never”, the code double-checks.
Deduplication at 0.92. Without it, the same fact gets stored again every time it comes up in conversation: “we use pnpm”, “project uses pnpm”, “pnpm is the package manager”. Duplicates waste the token budget and crowd out other memories. Any two sentences above roughly 0.92 cosine similarity are, for this model, almost always saying the same thing. I found that number by printing pairs and reading them, not by theory.
“Newer wording wins.” When a duplicate arrives, I overwrite the content with the new version. This is how corrections propagate. If the old memory said “deploy on Fridays” and the user now says “we deploy on Thursdays”, the two are similar enough to merge, and the newest version is the one that survives. Contradictions that are not similar enough to merge are handled by the consolidation job in Step 8.
Step 7: Share memory across agents with namespaces
In a multi-agent harness, “who can see what” is a design decision, not an afterthought. Namespaces are how I made that decision explicit.
A namespace is a label on every memory, like a folder name. Each agent writes to its own private namespace by default, and a small policy table decides which other namespaces it may read.
The policy in code:
// src/memory/policy.ts
export interface NamespacePolicy { read: string[]; write: string[]; }
export const POLICY: Record<string, NamespacePolicy> = {
planner: { read: ["agent:planner", "shared", "user"], write: ["agent:planner", "shared"] },
coder: { read: ["agent:coder", "shared", "user"], write: ["agent:coder", "shared"] },
reviewer: { read: ["agent:reviewer", "agent:coder", "shared", "user"], write: ["agent:reviewer"] },
};
export function readableNamespaces(agentId: string): string[] {
const p = POLICY[agentId];
if (!p) throw new Error(`No memory policy for agent "${agentId}"`);
return p.read;
}
export function assertCanWrite(agentId: string, namespace: string): void {
if (!POLICY[agentId]?.write.includes(namespace)) {
throw new Error(`Agent "${agentId}" may not write to "${namespace}"`);
}
}The reasoning behind each rule
Private by default. Each agent has its own working style and its own scratch knowledge. The coder’s note “the date utils file has a quirky timezone helper” is useful to the coder and noise to the planner. Keeping namespaces private by default keeps each agent’s context focused.
A shared namespace for team knowledge. Preferences and procedures apply to everyone: "use pnpm", "all new code needs tests". Those go to shared, so a fact learned by any agent benefits the whole team. This is where the multi-agent setup really pays off.
A read-only user namespace. Facts the user tells the harness directly ("I'm on Node 22", "never touch the legacy billing folder") get the highest trust. Only the user, through an explicit command, writes there. Agents can read it but cannot change it, so an agent cannot "remember" its way around a user rule.
The reviewer reads the coder’s memory. This was a deliberate asymmetry. A good reviewer needs to know why the coder made its choices (“used polling because the API has no webhooks”). But the reviewer should not write into the coder’s namespace, or the two agents could start rewriting each other’s knowledge. Read access across a boundary is cheap; write access across a boundary needs a strong reason.
Fail loudly. readableNamespaces throws when an agent has no policy. Silently defaulting to "read everything" is exactly the kind of bug that leaks context and is hard to notice.
Using it in the manager
The agent never chooses its own namespaces. The manager looks them up:
// inside the agent runner
const namespaces = readableNamespaces(agent.id);
const candidates = await memory.recall({ text: task, namespaces, limit: 40 });This is the principle of least privilege, the same idea behind file permissions and database roles: give each component only the access it needs. As your swarm grows, you will be glad that access rules are one table rather than if-statements spread across ten agents.
Step 8: Teach the system to forget
A memory system that only adds will eventually drown in its own history. After a month, my store had about 2,000 records, and recall quality was noticeably worse than in week one. The good memories were still there; they were simply outnumbered.
Forgetting is not a bug to minimize. It is a feature you have to build. I use two mechanisms, both running in a scheduled maintenance job rather than during user turns.
Mechanism 1: Prune what has faded
Every memory has a strength that combines how recently it was used, how important it is and how often it has been recalled. When strength falls below a threshold, the memory is deleted.
// src/memory/maintenance.ts (part 1)
import { recencyScore } from "./ranking";
import { MemoryRecord } from "./types";
const PRUNE_THRESHOLD = 0.05;
export function strength(r: MemoryRecord, now = Date.now()): number {
const recency = recencyScore(r.lastAccessedAt, r.kind, now);
const usageBoost = Math.min(1, Math.log1p(r.accessCount) / Math.log(20)); // 0..1, saturates near 20 uses
return recency * (0.5 + 0.5 * r.importance) * (0.5 + 0.5 * usageBoost);
}
export async function prune(store: MemoryStore, namespaces: string[]) {
const all = await store.listByNamespaces(namespaces);
const doomed = all.filter((r) => r.importance < 0.9 && strength(r) < PRUNE_THRESHOLD);
for (const r of doomed) await store.delete(r.id);
return doomed.length;
}How the formula behaves. Each factor is between 0 and 1, and multiplying them means any strong signal can keep a memory alive. A memory that has not been used in months but has importance 0.8 keeps a healthy importance factor. A memory used 20 times maxes out the usage factor. Only memories that are old, unimportant and rarely used drop below 0.05.
Math.log1p for usage. The logarithm gives diminishing returns: the jump from 0 to 5 uses matters a lot, the jump from 50 to 55 barely matters. Without it, one memory recalled hundreds of times would become immortal and unbalance everything.
The importance < 0.9 guard. Hard rules ("never run migrations against production") are never pruned automatically, however long they go unused. Some knowledge is valuable precisely because it is rarely needed.
Mechanism 2: Consolidate many episodes into one lesson
Episodes pile up fast. Over two weeks my coder stored eleven separate episodes that were all versions of the same lesson: “tests failed because the mock server wasn’t started”. Eleven memories, one idea.
Consolidation finds clusters of similar episodes and asks an LLM to summarize each cluster into a single durable fact. This is the closest thing my harness has to what sleep does for human memory: turning many specific experiences into one general piece of knowledge.
// src/memory/maintenance.ts (part 2)
const CLUSTER_THRESHOLD = 0.80;
const MIN_CLUSTER_SIZE = 3;
function clusterBySimilarity(records: MemoryRecord[]): MemoryRecord[][] {
const clusters: MemoryRecord[][] = [];
const used = new Set<string>();
for (const seed of records) {
if (used.has(seed.id)) continue;
const group = records.filter(
(r) => !used.has(r.id) && cosineSimilarity(seed.embedding, r.embedding) >= CLUSTER_THRESHOLD
);
group.forEach((r) => used.add(r.id));
if (group.length >= MIN_CLUSTER_SIZE) clusters.push(group);
}
return clusters;
}
export async function consolidate(memory: MemoryManager, store: MemoryStore, llm: LLM, namespace: string) {
const episodes = await store.listByNamespaces([namespace], ["episode"]);
for (const cluster of clusterBySimilarity(episodes)) {
const lesson = await llm.complete(
`These are related notes from past work:\n` +
cluster.map((r) => `- ${r.content}`).join("\n") +
`\n\nWrite ONE self-contained sentence capturing the durable lesson. ` +
`If the notes contradict each other, prefer the most recent. Reply with the sentence only.`
);
const importance = Math.min(1, Math.max(...cluster.map((r) => r.importance)) + 0.1);
await memory.remember({ kind: "fact", content: lesson.trim(), importance }, namespace, "consolidation");
for (const r of cluster) await store.delete(r.id);
}
}Understanding consolidation
The clustering is deliberately simple. It picks a seed memory and groups everything within 0.80 similarity of it. That threshold is lower than the 0.92 used for deduplication on purpose: duplicates say the same thing, cluster members are about the same thing. Real clustering algorithms (k-means, HDBSCAN) exist, but this greedy version is easy to read and debug, and it works well for a few hundred episodes.
MIN_CLUSTER_SIZE = 3. One or two similar episodes might be coincidence. Three or more suggests a pattern worth turning into a rule.
Episodes become facts. The output has kind fact, which has a much longer half-life than episode. The specific stories fade; the lesson stays.
Importance gets a small boost. A lesson that showed up three or more times is, by definition, recurring. Giving it slightly more importance than its strongest source reflects that.
Contradictions resolve toward recency. The prompt tells the summarizer to prefer the most recent note. Combined with the “newer wording wins” rule from Step 6, this keeps the store from holding two opposite beliefs at once.
Order matters: write before delete. The new consolidated memory is saved before the old episodes are removed. If the process crashes halfway, you end up with a duplicate, which the next run will merge. The opposite order could lose knowledge entirely.
I run prune and consolidate once a night through a scheduled job, plus at onSessionEnd for the namespaces touched during that session.
Putting it together: one full turn
We now have every piece. Here is how a single task flows through the harness, from the moment it arrives until its lessons are stored.
The read path (blue) runs before the model is called. The write path (purple) runs after, in the background.
And here is the runner that ties it all together:
// src/harness/runTurn.ts
import { randomUUID } from "node:crypto";
const MEMORY_BUDGET_TOKENS = 2000;
export async function runTurn(task: string, sessionHistory: Message[]): Promise<string> {
const turnId = randomUUID();
// 1-2. Route the task to the right agent
const agent = await router.pick(task);
// 3. Recall candidates from the namespaces this agent may read, then rank them
const candidates = await memory.recall({
text: task,
namespaces: readableNamespaces(agent.id),
limit: 40,
});
const ranked = rank(candidates);
// 4. Pack the best memories into the memory budget
const memoryCtx = buildMemoryContext(ranked, MEMORY_BUDGET_TOKENS);
await memory.touch(memoryCtx.included); // reinforce what we actually used
// 5. Call the model with system prompt + memory + recent turns + task
const reply = await agent.run({
system: agent.systemPrompt,
memory: memoryCtx.text,
history: trimHistory(sessionHistory, 2500),
task,
});
// 6. Fire the write path in the background; the user does not wait for it
hooks.emit("afterTurn", {
turnId,
agentId: agent.id,
userMessage: task,
agentReply: reply.text,
toolCalls: reply.toolCalls,
});
return reply.text;
}Reading the runner as a story
Read the function from top to bottom and it tells the whole story of this post:
- Route. Pick the specialist for the job.
- Recall. Embed the task and find memories that mean something similar, but only from namespaces this agent is allowed to read.
- Rank. Blend relevance, recency and importance so that fresh, important, on-topic memories rise to the top.
- Budget. Pack the winners into a fixed token allowance, skipping anything oversized, and refresh the memories we used.
- Think. The model sees the system prompt, a compact
<memory>block, the recent conversation and the task. - Remember. After the answer is sent, a hook extracts durable lessons, filters them, deduplicates them and stores them.
- Maintain. Separately, on a schedule, weak memories fade away and repeated episodes become lasting facts.
Notice what the agent itself does not know about: SQLite, vectors, half-lives, deduplication thresholds. To the agent, memory is simply a paragraph of helpful background that appears in its prompt. That is the sign of a good abstraction. All the complexity is real, but it lives in one place.
A tip for debugging
Log memoryCtx.included with each memory's relevance, recency, importance and final score on every turn. When the agent does something strange, the first question is always "what did it remember?" Having that answer in a log line turned hour-long debugging sessions into five-minute ones.
Lessons learned the hard way
Building this by hand taught me more than any library’s documentation could have. These are the mistakes I would warn my past self about.
1. Memory quality is decided at write time, not read time. I spent days tuning ranking weights before realizing the real problem was what I was storing. Garbage in the store cannot be ranked into gold. A strict extraction prompt and an importance floor did more for recall quality than any formula change.
2. Store sentences, not fragments. Every memory must make sense when read alone, weeks later, by an agent with no context. “Use the second approach” is worthless. “Retries use exponential backoff starting at 200 ms” is gold. If you change only one thing in your own system, change this.
3. Count tokens, never items. “Top 10 memories” is a bug waiting to happen. One oversized memory can blow your context window. Budget in tokens and cap the size of each individual memory.
4. Recency must mean last used, not last created. Otherwise your most important long-standing facts slowly fade out of every prompt, precisely because they have been true for a long time.
5. Without deduplication, the loudest fact wins. A fact mentioned in ten conversations gets ten copies, which then occupy ten slots in the prompt and push out everything else. Merge near-duplicates on write.
6. Memory needs a way to be wrong. Old memories will contradict new reality. Tell the model in the prompt that current instructions win, let newer wording overwrite older wording on merge, and resolve clusters toward the most recent note.
7. Never persist secrets. It is very easy for an API key in a tool output to become a “fact” that is then injected into every future prompt. Filter in the prompt and again in code.
8. Make memory observable. Log what was recalled, its scores, and what was written on every turn. When an agent misbehaves, “what did it remember?” is almost always the first question, and you want the answer in seconds.
9. Thresholds are empirical. The numbers in this post (0.30 relevance floor, 0.92 duplicate threshold, 0.80 cluster threshold) were found by printing real scores and reading them. They depend on your embedding model. Copy the method, not the numbers.
10. Start brute force. A for loop over a few thousand vectors is fast enough. I almost introduced a vector database on day one, and I would have spent that week on infrastructure instead of learning how memory actually behaves.
Hand-rolled vs. using a framework
Would I recommend everyone build their own memory layer for production? No. Mature frameworks handle edge cases, scale and concurrency that this design ignores. But I would recommend building it once. After this project, when I read the documentation of any memory or RAG framework, I can map every option to a concept I have implemented myself: namespaces, half-lives, consolidation, token budgets. That understanding is what makes you good at choosing and configuring tools, rather than just installing them.
Conclusion and what’s next
An agent’s memory is not a feature of the model. It is a system you build around the model, and it comes down to answering four questions well:
- What is worth remembering? Extract durable, self-contained facts after each turn; filter noise and secrets.
- What is relevant right now? Embed, compare by meaning, and rank by relevance, recency and importance.
- How much can we afford to show? Pack the best memories into a fixed token budget.
- What should we let go? Prune what has faded and consolidate repeated experiences into lasting lessons.
Wiring each of those by hand turned “memory” from a black box into a set of small, understandable pieces: a record type, a store, an embedder, a ranking function, a packer, a hook and a maintenance job. None of them is more than a hundred lines of TypeScript. Together, they turned my team of forgetful contractors into something that actually learns from working on a project.
The next pieces I am building in the harness:
- Hybrid search. Combining embeddings with keyword search (BM25) so exact identifiers like function names and error codes are recalled reliably.
- A memory inspector. A small UI to browse, edit and delete memories by namespace, because the user should always be able to see and correct what the agents believe.
- Evaluation. A fixed set of tasks with known “correct” memories, so that every change to thresholds or prompts can be measured instead of guessed.
If you are building your own agent harness, I hope this saves you a few of the evenings it cost me. Start with the MemoryRecord type, keep the store boring, and spend your energy on what gets written.
