Context Engineering
1. What is Context Engineering ?
Context engineering is the strategy of designing, structuring and optimizing the context that is being provided to an LLM (everything that lands in the model’s input, not just the prompt itself) to generate responses. It is about finding the smallest set of high-signal tokens that make the desired behaviour most likely.
2. Why is Context Engineering a Big Deal ?
A context window is finite and it is not a neural storage space. It has diminishing returns.
Context rot: As the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.
Models suffer from “lost in the middle” effects (info buried in the middle of long context gets underweighted) and from dilution (relevant signal gets drowned by irrelevant filler).
So, even before you hit a hard token limit, an agent can be getting less value out of each additional token you stuff in. Anthropic’s 2026 Agentic Coding Trends Report found teams with well-maintained context practices complete tasks 55% faster and produce 40% fewer errors than teams without.
3. What Components Fill Up a Context Window ?
Every inference call is built from roughly these pieces and each has its own failure modes:
i. System prompt / instructions: Sets role, constraints, output format. The Anthropic guidance here is to be “thoughtful and keep your context informative, yet tight”; not to write a bloated instruction manual covering every edge case.
ii. Tools: This is where things go wrong most often in practice. A common failure mode is bloated tool sets that cover too much functionality or lead to ambiguous decision points about which tool to use. Tool descriptions should be non-overlapping, and parameters unambiguous.
iii. Examples / few shots: Curated, diverse examples that show the desired behavior.
iv. Message history / retrieved context: The part that grows unboundedly over a long session — conversation turns, tool call results, retrieved documents.
4. Core Techniques for Agent Context Engineering
4.1. Write: store context outside the active window entirely in scratchpads (can be implemented using a tool call that writes to a file or in a runtime state object), memory files, running plans. The model doesn’t have to re-derive facts from scratch each turn. This is the “note-taking” pattern. Anthropic’s multi-agent researcher does exactly this. The lead agent saves its plan to memory before starting work, specifically because if the context window exceeds its limit and gets truncated, the plan needs to survive.
Saving it outside the context window to help an agent perform a task.4.2. Select: retrieval. Pull in only what’s relevant right now rather than dumping a whole knowledge base in. This is RAG’s job, but done well it’s targeted and re-ranked, not a naive top-K dump. Memory selection is genuinely hard. Selection from scratchpad depends on how it is implemented. If it is a tool, a simple tool call can be made. If it is a part of the agent’s runtime state, the developer can choose what parts of the state to expose. With respect to memory, agents might select few-shot examples (episodic memories) for examples of desired behavior, instructions (procedural memories) to steer behavior, or facts (semantic memories) for task-relevant context.
Pulling it into the context window to help an agent perform a task.4.3. Compress: this is compaction. Compaction takes a conversation nearing its context limit, summarizes it, and reinitializes the session with that summary so the agent can continue with minimal degradation. The hard part is what to keep vs. discard. Overly aggressive compaction can silently lose context whose importance only becomes obvious later. Anthropic’s own tooling lets you customize the compaction prompt itself, because the default summarization strategy won’t preserve the right things for every workload.
Context summarization: typically uses an LLM to distill the most relevant pieces of context.
Context trimming: filter or prune context.
Retaining only the tokens required to perform a task.4.4. Isolate: split work across separate context windows entirely, usually via sub-agents. Anthropic’s multi-agent researcher found that many agents with isolated contexts outperformed a single agent, because each sub-agent’s window could be devoted to a narrower sub-task, exploring different aspects in parallel. The catch: this costs tokens. Anthropic reported up to 15× more token usage than a single chat. Plus real overhead in planning and coordinating the sub-agents. Agent’s runtime state object can be great way to isolate context.
Splitting it up to help an agent perform a task.5. Problems Associated with Context (How Long Context Fails)
5.1. Context Poisoning
When a hallucination or error gets into the context and then gets referenced repeatedly, compounding over time. The agent doesn’t just make a mistake once, it treats its own earlier mistake as established fact and keeps building on it.
DeepMind’s Gemini 2.5 technical report gave a vivid example from their Pokémon-playing agent. An especially egregious form of this issue took place where many parts of the context — goals, summary were “poisoned” with misinformation about the game state, and the agent kept acting on that false belief. In coding agents, this often looks like: the agent misdiagnoses a bug, writes that wrong diagnosis into a NOTES.md or its own running summary, and every subsequent step treats the wrong diagnosis as ground truth. A wrong note hardens into truth, often written with good intentions as the agent tries to track its own progress. In multi-agent pipelines it's especially dangerous when Agent A's output feeds Agent B, which feeds Agent C, a single poisoned fact in step three can produce a subtly wrong final output that looks completely plausible and nothing throws an error, because nothing technically crashed.
Primary fix: structured external storage/checkpointing rather than letting the agent’s self-written summaries be the only record. Plus the ability to roll back to a known-good state when corruption is detected, rather than only ever appending forward.
5.2. Context Distraction
When the sheer volume of accumulated history causes the model to over-rely on repeating past patterns instead of reasoning fresh, it starts pattern-matching against its own history rather than synthesizing a new plan. Instead of using its training to develop new strategies, the agent became fixated on repeating past actions from its extensive context history.
Get pranamya’s stories in your inbox
Join Medium for free to get updates from this writer.
This kicks in well before the hard context limit. The Gemini Pokémon team found the tendency emerged as the context grew significantly beyond 100k tokens. The agent showed a tendency toward favoring repeating actions from its vast history rather than synthesizing novel plans, and it’s model-dependent: a Databricks study found that model correctness began to fall around 32k tokens for Llama 3.1 405b, and earlier for smaller models. In practice this shows up as an agent stuck in a retry loop, repeating a failed approach because that approach is now the most prominent thing in its window.
Primary fix: compaction/summarization to keep the working history lean. This is why summarization matters even when you’re nowhere near the token limit.
5.3. Context Confusion
When superfluous, irrelevant content in the context gets used by the model anyway, degrading the response. Most commonly, picking the wrong tool because too many tool definitions are competing for attention. Studies show accuracy degrades measurably when agents have more tools, with the failure rate worsening on smaller models and growing non-linearly as the tool count rises past thirty. It also shows up when a broad search or directory dump pulls in look-alike content — test fixtures, dead implementations, mocks, and similarly named functions from unrelated modules, all of which look plausible and the model can’t resist attending to it. Notably, models weight the window heavily. They don’t treat context as optional reference material, they reliably incorporate it even when it’s irrelevant, and a single topically-related distractor is enough to measurably drop performance.
Primary fix: just-in-time, scoped retrieval. Give the agent a smaller, curated tool/document set for the task at hand rather than everything that might vaguely apply.
5.4. Context Clash
When new information or instructions entering the context directly contradict something already there, leaving the model stuck between incompatible assumptions rather than just ignoring noise (that’s confusion) or over-anchoring on old patterns (that’s distraction). This is when you accrue new information and tools in your context that conflicts with other information in the prompt. It’s especially common when you pull in third-party tools you didn’t write. When you connect to MCP tools you didn’t create, there’s a greater chance their descriptions and instructions clash with the rest of your prompt.
It also has a subtler, more insidious form: once a model commits to a wrong read on a problem, contradictory evidence doesn’t overturn it cleanly — models commit early to a reading and struggle to let go of it; tasks they solve cleanly when handed everything in one pass, they fumble once the same information arrives muddled across turns. Breunig calls this one the most damaging of the four: when an agent takes a wrong turn midway through a task, the incorrect reasoning remains in the context window and continues to influence all subsequent steps.
Primary fix: sub-agent isolation, so one task’s noisy or contradictory information never contaminates another’s context. Plus being deliberate about vetting third-party tool descriptions before they enter your prompt.
6. Problems with Multi-session / long-horizon agents
What happens when a task spans multiple context windows entirely (e.g., a coding agent working for days)?
Each new session begins with zero memory of what came before; like a software project staffed by engineers working shifts, where each new engineer arrives with no memory of the previous shift. Anthropic found that compaction alone isn’t sufficient here. Even a frontier model given only a high-level prompt like “build a clone of claude.ai” would try to one-shot the whole thing, run out of context mid-implementation, and leave the next session to guess at a half-finished, undocumented feature. The fix that worked was having an “initializer” pass write a durable, external spec (in one case, 200+ discrete features marked pass/fail) that future sessions could pick up cleanly. Essentially turning the filesystem into the persistent memory layer instead of relying on the model’s window.
7. Positional bias — “lost in the middle”
Even within a technically valid context length, where information sits matters. The foundational “Lost in the Middle” research found LLM performance follows a U-shaped curve: models use information at the beginning and end of the context window effectively, but accuracy drops by 30% or more when the relevant fact sits in the middle. The mechanism is architectural. Positional encoding biases make transformers attend more strongly to the edges of the input. Practical implication: don’t bury the one document that matters at position 40 of 80 retrieved chunks; put high-value context near the start or end, and if you can’t control that, rerank so what matters most is closer to an edge.
8. Prompt caching (the cost/latency lever)
Separate from what you put in context is how expensively you reprocess it. Prompt caching stores the computed key-value representations for a stable prefix (system prompt, tool schemas, a long reference document) so repeat calls skip reprocessing it. Anthropic’s version uses developer-controlled cache_control breakpoints (up to four per request) with 5-minute or 1-hour TTLs, and cache reads run at roughly a tenth the price of fresh input tokens. This is directly a context-engineering concern, not just a billing detail: it changes your incentives around where you put dynamic vs. static content in the prompt (static stuff belongs in a stable prefix, and even small differences — tool ordering, timestamps, non-deterministic JSON key ordering — silently break the cache), and it means aggressive context-trimming/compaction has a real cost trade-off, since restructuring history on every call defeats caching entirely.
9. Context Observability and Evaluation
You largely can’t manage what you don’t measure. This means: token attribution (how many tokens is your system prompt vs. tool defs vs. history actually consuming), needle-in-a-haystack-style benchmarking to find your specific model’s “distraction ceiling” (the point where added context starts hurting rather than helping — this varies by model and task type), and treating context-engineering changes (a new compaction prompt, a new retrieval threshold) as things that need regression testing against real traffic, the same way you’d test a code change — over-compressing context to save tokens can quietly degrade answer quality and trigger more retries than it saved.
10. Security: context as an attack surface
Anything that lands in context is something an adversary can potentially manipulate. This is the basis of prompt injection via retrieved documents, tool outputs, or third-party MCP servers (a malicious or careless tool description can smuggle in instructions the model will follow). Related is the “excessive agency” problem. Giving an agent more tool permissions/autonomy than the task strictly needs widens the blast radius if any of the context it’s reasoning over turns out to be poisoned or adversarial. This is a reason to scope tool access tightly per-task/per-subagent rather than granting one agent broad access to everything.
11. Where research is heading
The most interesting frontier work asks: what if the agent manages its own context budget instead of an engineer hand-tuning it? The ACE (Agentic Context Engineering) pattern, from a Stanford/SambaNova/Berkeley paper presented at ICLR 2026, splits this into three roles — a Generator that produces reasoning trajectories, a Reflector that distills lessons from what worked and failed, and a Curator that folds those lessons into a structured, evolving “playbook” rather than rewriting one static system prompt from scratch each time. The key insight is that incremental, additive updates to context beat wholesale rewrites, representing context as itemized bullets you append to, rather than a blob you regenerate. It’s promising but still early. Most teams get more value first from a disciplined, manually-set context budget before reaching for a self-curating system.
