Member-only story
AI Agents in Production — AI Agent Drift: Harness Fingerprint, Trajectory Distributions, and Bisection by Replay
A step budget stops one run, while drift shifts the distribution. A harness fingerprint, a frozen holdout, and bisection name the component
The eval passes green, the model is pinned to a dated snapshot, and the agent’s code hasn’t changed in a month. Yet the median step count keeps climbing, cost per task jumps, and at stand-up nobody can name the cause. This scene is a composite built for illustration, but its mechanism is documented.
Three changes to a coding agent’s harness reached production in March and April 2026: a lower default reasoning effort, a bug in clearing the thinking history, and a new system prompt instruction that, combined with other prompt changes, lowered quality. Each hit a different slice of traffic at a different time, so the whole thing looked like broad, inconsistent degradation with no single culprit. The API and the inference layer, meanwhile, stayed untouched (Anthropic postmortem of April 23, 2026).
The texts on agent drift that I read usually end with a catalog of change sources and a checklist. I found no method in them for the week when three components change at once. After…
