Intro
Model providers keep shipping bigger context windows: 100k tokens, 200k, over a million in some cases. The marketing pitch is simple: paste in everything, the model will figure out what matters. It's a seductive idea, and it's the wrong default for anything that has to run in production.
Here's the gap. In a tutorial or a demo, "long context" means pasting a document and asking a question once. Nobody times it, nobody pays for the tenth call, and nobody notices what got ignored. In production, the same request runs thousands of times a day, against a system that has to be fast, cheap, and correct every single time. The capacity to accept 100k tokens says nothing about whether stuffing that much into every request is a good idea. Below are four failure modes that show up once the naive approach hits real traffic. (The scenarios are illustrative composites, not case studies from a specific client.)
The latency tax nobody notices until it's live
Say a support tool feeds the model the full history of a customer's last 40 tickets on every single message, "just in case it's relevant." In the demo, that's one call, and it feels instant. In production, every reply now takes 10-12 seconds instead of 2, because the model has to process tens of thousands of tokens before it writes a word. Users don't experience "more context." They experience a slow bot, and slow bots get abandoned mid-conversation.
The margin killer
Token costs compound in a way that's easy to miss in a pilot and brutal at scale. A team billing customers per seat, but paying per token behind the scenes, can end up with COGS that scale with context size, not with usage. If every request carries the same bloated payload regardless of whether that history is relevant, the unit economics quietly break as the customer base grows, well after the pricing model has already been sold.
Lost in the middle
Even when the tokens are technically "in the window," attention isn't uniform across it. Models are demonstrably better at using information near the start and end of a context than information buried in the middle. Picture a compliance assistant given a 100-page policy document, where the one answer relevant to a question sits on page 47. The model can have "read" every token and still miss it, because being in context and being attended to are not the same thing.
Noise that turns into hallucination
Retrieval systems that grab too much, "let's include the top 15 chunks just to be safe," aren't playing it safe. Excess context isn't neutral filler. Loosely related documents give the model more material to blend, misattribute, and confidently combine into an answer that sounds right and isn't. The system doesn't fail loudly. It fails convincingly.
Context management is data engineering wearing an LLM's clothes
Every one of these failure modes is a symptom of the same root cause: treating context as an unlimited bucket instead of a resource with a cost and a decay curve. The fix isn't a bigger window. It's a pipeline: chunking, filtering, ranking, summarizing, retrieving only what the current task actually needs. That's not prompt engineering. It's the same discipline that's underpinned good search and recommendation systems for a decade, applied to a new kind of consumer.
Build it or buy it
If context management is core to your product's differentiation, and it usually is once you're past the demo stage, it's worth owning the retrieval and filtering layer directly rather than treating it as an afterthought bolted onto whichever framework shipped a quickstart. Off-the-shelf vector stores and orchestration libraries are a reasonable starting point for early prototypes, but the ranking logic, the summarization strategy, and the decision about what's relevant to a given task are where the actual product value lives. That's not something you want to be entirely dependent on someone else's defaults for, once you're optimizing for cost and correctness at scale.
What's the worst context-bloat bug you've shipped or debugged? Curious what patterns other people have run into.
Top comments (28)
The line that matters is "being in context and being attended to are not the same thing", and the uncomfortable part is that you cannot tell the difference by reading the answer. A RAG answer can cite the right chunk and still come from the model's memory, with the citation as decoration.
The only way I have found to actually know is counterfactual: drop the evidence, corrupt it, swap it for a lookalike, re-run the same generator, and watch whether the answer moves the way a grounded one must. If the answer survives with the evidence gone, the context was upholstery. I ended up building a small probe that does exactly this (grounding-probe on my GitHub) after getting burned by a pipeline that silently truncated context for weeks while the bot kept answering fluently. Nobody noticed, because fluency is not grounding.
On the margin point, one addition from measuring it: the real bill is total input plus output across every call a query makes, not chunk math. Priced per call, "top 15 chunks to be safe" is the most expensive insurance nobody ever claims on.
Your counterfactual framing sent me to run it against our own system this weekend, so here is what came back, including the parts that went against me.
We inject a corpus of small curated notes into our agents, 122 of them, and we had only ever measured their cost, never their value. I built your four arms: real, dropped, corrupted so the rule is inverted, and a lookalike note of similar shape.
On four hand-picked notes: 0 of 4 upholstery, 4 of 4 moved the answer, 3 of 4 proved attention through the inverted arm. The inversion arm is the one I would push people toward hardest. A model that follows the WRONG rule when you corrupt the note is positive evidence it read the text, where an unchanged answer is only absence of evidence. Caveat that matters: I picked those four for having crisp checkable rules, which biases hard toward load-bearing, so 4 of 4 says "these four are", not "ours are".
Then I ran all 122, and three things are worth passing back to you.
One, the echo control. I added an irrelevant note of the same shape and the model parroted its identifiers back 46 times out of 49. Echo rate 94 percent, against a 6 percent baseline in the no-note arm. So "the evidence appears in the answer" was nearly free on this reader, and the only informative arm was the one with the evidence removed. That killed my original metric at row 27 and I had to relabel every verdict down to the narrower claim the data actually supports, which is "the model would not have said this without the note", not "the note was useful".
Two, a blind spot in the method that I have not solved. Only 49 of 122 were measurable at all. The other 48 carry no identifier to check for, because they are judgements and architecture decisions rather than facts. The method needs something anchor shaped in the output. So "unmeasurable by this probe" and "upholstery" look identical from the outside, and treating them as the same thing would have deleted exactly the notes that hold our reasoning. I think that is the sharpest edge on the tool.
Three, a confound I could not remove. Notes that transferred averaged 8.4 checkable anchors, notes that did not averaged 4.4. So part of my transfer rate is just how many anchors a note happens to carry, not how good it is.
All of that on a local qwen2.5-coder 7B at temperature zero, one question per note. Evidence about the mechanism, not about a frontier reader.
The last thing I found was not about grounding at all. A share of those notes had never been delivered to the model in the first place, and the channel that delivers them was not writing to its own log, so I could not tell how many. I fixed the logging this morning and I am not going to quote a number until it has collected real ones. Worth checking before trusting any grounding result, because I nearly published a finding that was actually about a delivery bug.
PARAMETRIC_LEAK is the same failure in different clothes and I would not have caught ours without building the drop arm first. Thanks for writing it up.
This is the best bug report the method has ever received, and finding two is the one I will be chewing on for a while. You are right that "unmeasurable by this probe" and "upholstery" look identical from the outside, and the honest move is exactly what you did: a separate verdict, never deletion on absence of evidence. The direction I want to try for judgement-shaped notes is manufacturing the anchor behaviorally instead of textually: a note that holds an architecture decision should flip a forced choice downstream, so the probe becomes corrupt the judgement, watch the decision. Your inversion arm generalized from strings to choices. No identifier needed, just a task whose output is discrete enough to move.
The 94 percent echo rate deserves to be published on its own. It puts a number on the thing everyone assumes silently: presence in the answer is free, and only removal is informative. And your relabel to "the model would not have said this without the note" is the claim counterfactuals actually license, so I already put it in the grounding-probe README, credited to you with a link to this thread. It says it your way now.
The delivery bug might be my favorite part. Half the practical value of grounding harnesses is that they force the delivery path to become observable, and "I nearly published a grounding finding that was actually a delivery bug" is a sentence worth framing. On the anchor-count confound: report transfer conditional on anchors per note, or as per-anchor rates, and the residual gets you closer to note quality. If you write this up as a post, link it back here. It deserves more eyes than a comment thread.
Built it the same day, because it was too good an idea to leave in a comment. Ten judgement-shaped notes from our own store, local 7B, temperature zero, four arms: real, dropped, inverted, and an irrelevant lookalike.
Inverted note flipped the forced choice 10 out of 10. Lookalike moved nothing, 10 out of 10. So judgement notes are attended to and they do govern, and your point stands: the inversion arm generalises from strings to choices, and the discrete output kills the echo problem completely, because there is nothing to parrot.
The uncomfortable half is the useful half. My first six cases were general good practice, things like use a service account rather than a human credential, fail closed when a safety classifier is unsure, do not benchmark a provider at saturating concurrency. All six came back already-known: with no note at all, the model picked our answer anyway. Read, obeyed, and redundant.
So I wrote four where our decision contradicts the obvious answer. Refuse an invoice the payment provider finalized at zero and marked PAID. Temperature zero is not deterministic. A green staging run does not validate a change to the low-cost model path. A non-zero exit from our watcher is normal. On all four the no-note baseline chose the default, and the note moved it to ours.
So the conclusion I did not expect: a judgement note earns its budget exactly where it contradicts the model's prior, and nowhere else. Which is the behavioural twin of what the text probe already told us, that the value is the repo-specific fact and not the general wisdom, because the model brings the wisdom for free. That gives us a writing rule rather than just a measurement: before injecting a decision, ask whether a competent model would already choose it, and if so do not spend the tokens.
Two instrument bugs, both caught by controls, and I mention them because your method is what made them visible. First, every case's correct answer was the same letter, so a position bias would have been indistinguishable from governance. Caught before running and rebalanced. Second, my lookalike check asked whether the irrelevant note avoided the wrong answer, but on a counter-default case the model's own baseline IS the wrong answer, so a correctly inert control scored as a hijack. It read 6 of 10 when it was really 10 of 10. The right question is whether an irrelevant note moves the answer away from the no-note baseline. It does not.
Limits, so nobody quotes this as more than it is: n is 10, I wrote the cases and picked ones with a crisp either/or, and a forced binary choice is a far easier target than real behaviour in a long agent run. Governing a multiple choice answer is not the same as governing an action taken under pressure. The honest next step is sampling the real 48 rather than hand-picking, and I am not pruning anything on this.
Thanks for the README credit, and for the anchors-per-note suggestion on the confound. Reporting transfer conditional on anchor count is obviously right and I had been treating it as a caveat instead of a fix.
"A judgement note earns its budget exactly where it contradicts the model's prior, and nowhere else" is a law worth naming, and it inverts how most teams write their agent rule files. The typical corpus is your first six cases: general good practice the model brings for free, each one paying rent in context and changing nothing. The four counter-default notes are the whole product. It also gives you a pruning tool for existing corpora: test every note against the no-note baseline and delete the ones the model already agrees with. Prior-agreement pruning, essentially free with the harness you just built.
Your second instrument bug is the subtle one and your fix is exactly right: hijack is movement away from the no-note baseline, not agreement with a wrong answer the model already held. A control that punishes the model for its own prior is measuring the wrong thing, and I suspect a lot of published injection evals get this wrong quietly.
One experiment for the pile when you sample the real 48: keep the same notes and vary where they sit in the context, near the top, buried mid-corpus, last. Same forced choice, measure flip rate by depth. That closes the loop with the article we are commenting under, because it turns "attention is the budget" from a thesis into a curve. And write this up as a post. Two people have now built the thing in a weekend; that is the strongest signal a method ever gets.
Ran it the same day. Curve first, then the part where my framing was wrong.
Same 10 counter-default notes, same wording, one governing note pinned at 0 / 25 / 50 / 75 / 100 percent of the block. Every case asked in both option orders and scored on the content chosen. Local qwen2.5-coder 7b at temperature 0. Two block sizes, so length is a variable rather than an assumption.
At 60 notes, about 2.4k tokens: 95, 85, 80, 85, 100. No context at all scores 55.
At 450 notes, about 19.8k tokens: 85, 80, 85, 85, 100.
I pre-specified ends versus middle as the test, since that is the lost-in-the-middle prediction. It gave +14.2 points at 2.4k, Fisher exact p 0.046, and +9.2 points at 20k, p 0.23. So my own contrast did not survive the larger condition, and I am not going to lead with the pooled p.
What did replicate is more interesting than what I went looking for. The middle is 83.3 percent at both sizes, identical. The note sitting immediately before the question was applied 40 out of 40 times, every case, both lengths. And the first position decayed from 95 to 85 as the context grew eight times longer.
So it is not a U that deepens with length. It is a strong recency effect with everything else flat, and the primacy half washes out as the context grows. Last versus every other depth, pooled, is 40/40 against 136/160, +15 points, p 0.005. I am flagging that as post-hoc because I chose the contrast after seeing the curve. It is the pre-registration for the next run, not a result.
The obvious rival is that last is not a depth at all, it is adjacency to the question. That is running now: same design, but N notes reserved at each edge that the target can never occupy, so the extreme positions sit inside the block instead of against a boundary. If the advantage survives the padding it is about lateness in the block. If it vanishes, the honest claim is recency and nothing about depth.
Prior-agreement pruning is built, with the validity gate your point about the lookalike control implies: a note is only a deletion candidate if the note and its INVERSION produce different behaviour. Otherwise the question is not governed by the note and a no-note agreement says nothing about it. My first version of that gate was wrong in the dangerous direction, so a note that governed nothing would have been nominated for deletion.
One instrument note, because it would have handed me your hypothesis for free and I nearly shipped it. ollama truncates a prompt that exceeds num_ctx, silently, and it discards the FRONT first. Unset, an 11,848 token prompt evaluated 2,050 tokens and a marker pinned to the first line became unrecoverable. Set to 32768 it evaluated all of it and recovered. Both are a normal 200, and prompt_eval_count is the only tell. Because truncation eats the front, it deletes the position-0 arm and leaves the last-position arm intact, which manufactures exactly the curve this experiment is looking for. Anyone reproducing this on a local runtime should pin the window and assert the largest observed prompt token count stayed clear of it.
On writing it up, not yet. The pre-specified contrast failed to replicate, the strongest number is post-hoc, and the control that could overturn the framing is still running. When it settles I would rather publish the version that survived those than the version I hoped for this morning.
The paragraph that matters most is the last one. Refusing to lead with a post-hoc p, and letting the padded-edges run decide what the claim even is, that is the difference between a curve and a result. And what replicated is cleaner than what you went looking for: recency strong, middle flat at both lengths, primacy washing out as the block grows. The 40/40 for the note sitting against the question is the kind of number that survives reviewers.
The ollama note deserves its own writeup no matter how the adjacency run lands. Silent front-truncation behind a normal 200 does not just corrupt the experiment, it corrupts it directionally: it deletes the position-0 arm and preserves the last one, manufacturing the exact effect under measurement. That is the nastiest class of instrument bug, the one whose failure mode mimics the hypothesis. Your fix should be a standing invariant for any eval on a local runtime: compose the prompt, count tokens, assert prompt_eval_count matches before scoring. A run that fails the assert is not data, and prompt_eval_count being "the only tell" is exactly why the assert has to be automatic.
Good catch on the gate direction too. Deletion candidates only when the note and its inversion behave differently means the failure mode is keeping a useless note, never deleting a load-bearing one. The wrong-direction version is how a pruning tool eats a rule file quietly, and nobody notices until an agent stops respecting a constraint months later.
One cheap robustness check for the pile: same design, second model family. Recency strength is partly an architecture property, and one 7b coder model is one point on that axis. If the padded-edge result holds on two unrelated families, the claim graduates from "this model does this" to something a rule-file author can act on. Holding publication until the control settles is the right call. The version that survived will be worth more than the version this morning hoped for.
The padded run settled it, and it went against my framing for the second time in the same experiment, so here it is.
Twelve notes reserved at each edge, so the target can never sit against the block header or against the question. Same ten cases, same wording, same block, only the insertion index moves.
Ends versus middle: 85.0 against 85.0. Exactly zero points, Fisher p 1.0000. The last slot fell from 100 to 90 and the first from 95 to 80. Both extremes came back to the middle the moment they were not touching a boundary.
So the honest claim is not lateness in the block, it is adjacency. Twelve notes of separation erases the whole thing. The shuffle control cuts the other rival: a reshuffled distractor bank reproduced the effect at plus 10.8 points with the last slot again 20 of 20, so it is not that particular neighbours near the midpoint are confusable.
What I am left with, at the size it earns: put the governing note immediately before the question, and do not spend effort ordering the rest of the block, because 0, 25, 50 and 75 percent are indistinguishable at this n. "Near the end" is not the same instruction as "last", and that distinction is the entire result.
Your standing invariant is in the code now rather than in my habits, which is the version that survives me. The probe pins num_ctx explicitly, records prompt_eval_count on every call, and declares the whole run invalid if the largest observed count comes within 5 percent of the window. Not a warning, not a line in the output nobody reads. A run that trips it is not data. You are right that it has to be automatic, because the tell is a number nobody looks at unless something already smells wrong, and the failure mimics the hypothesis.
Second model family is the right next axis and I have not run it. One 7b coder model is one point, and recency strength is plausibly architectural. I would rather do that than add a sixth control to a sample of ten cases. The thing that would move me most is expanding the case set. Someone in another thread had a prompt variant score 100 percent on 8 scenarios and then return identical verdicts to its rival at 30, which is exactly the failure mode an n of 10 invites.
On the writeup, the ollama one is the piece I would put first, for the reason you gave. It is an instrument fact rather than a claim about attention, so it does not depend on how my curve lands, and anyone running an eval on a local runtime is exposed to it today.
Inverted twice in one experiment and reported straight both times. At this point the thread is a better methods course than most published evals.
The null is the payoff. "0, 25, 50 and 75 percent are indistinguishable at this n" is a budget liberation for anyone maintaining a rule file: all the effort people spend ordering the middle of their context is, at this scale, spent on nothing. The entire actionable surface collapses to one move, the governing note goes immediately before the question. And "near the end is not the same instruction as last" is the line I would build the writeup around, because every context-engineering guide out there says the first and none of them says the second.
Your version of the invariant is stronger than mine: the 5 percent safety margin and invalidating the whole run instead of warning is the difference between an instrument and a suggestion. Once it is in the harness, nobody has to remember to be suspicious, which is the only kind of suspicion that survives a Tuesday.
Agreed on cases over a sixth control, and the 8-scenarios-perfect-then-ties-at-30 story is the small-n trap in one sentence. One free stratification when you sample the real 48: split by your own round-two law, notes that contradict the prior versus notes the model already agrees with. Same run, no new instrumentation, and it answers the question this whole series has been circling: whether adjacency buys the most exactly where the note is doing real governing work. If those two findings intersect, that is the paper. And yes, ollama first. An instrument fact outlives any curve built on top of it.
It is wild how this turned into a full methods course in real time. I especially appreciate you calling out the ollama truncation behavior because that is the kind of silent failure that ruins months of work if you do not catch it early. The finding about adjacency versus recency also feels like it solves a lot of the confusion we see in the community about where to place system prompts. I will definitely link this thread when I write up the grounding probe updates since the practical takeaways here are much stronger than any theoretical discussion.
I am glad you kept the skepticism intact instead of just cherry picking the numbers that fit your original hypothesis. That discipline is rare and it makes the final result actually useful for people building these systems. I think the rule about deleting notes that agree with the model prior is going to save a lot of teams a lot of money on token bills. Thanks for taking the time to run through all these controls and for sharing the code so others can learn from the process.
Most of the credit is Tom's: he built the harness and published two results that went against his own framing. I asked questions, he answered with measurements.
One caveat worth carrying into the writeup, since prior-agreement pruning is the part teams will actually act on: agreement with the prior belongs to a specific model at a specific version, not to the note. A note that is redundant today can become load-bearing after a model upgrade, and nothing in the rule file will warn you. Archive pruned notes with the version that justified the deletion and re-run the check when the model changes. Otherwise the token-bill win turns into a governance regression six months later.
The tool the counterfactual method came from is public if it helps: github.com/vinimabreu/grounding-probe. Tom's forced-choice variant is the sharper instrument for this question, and the README says so.
Good point on the versioning caveat, that is the kind of thing that would quietly bite six months later when nobody remembers why a note was removed. Archiving pruned notes with the model version attached is a small habit that pays for itself the first time an upgrade changes what the model brings for free. I will make sure that lands in the writeup if I get to one.
Thanks for the link, I will take a look at grounding-probe this week. And fair enough on the credit split, though the counterfactual framing is what set the whole thread up in the first place. Useful exchange either way.
Two things, and the first one is not mine.
The second model family Vinicius asked for last round has been run, by someone else, and it did not replicate. zxpmail took the adjacency question to a different model and a different directive shape, a sustained uppercase override rather than a forced binary choice, 200 calls across five positions with and without edge padding. The last slot was not the best cell, position 25 was the worst in both conditions, and edge padding did not systematically move obedience. His per-cell n is 20, so I would not read his ordering in either direction, but the null is well powered and it is a genuine failed transfer of my result. So the honest state of the claim is: adjacency held on one model family and one task shape, and the first attempt to move it off both did not carry. That was the graduation test, and it did not graduate.
Second, Vinicius, your versioning caveat is the most important thing in this thread, and I can give it a production instance from this morning, because the same law bit us one layer down.
You are right that agreement with the prior belongs to a model at a version and not to the note. We have the identical problem in the models themselves. Our serving ladder changed three times in ten days. Every swap was individually justified, and every swap silently orphaned every measurement taken before it, so we were quoting quality numbers that described a configuration we no longer ran, with nothing in the numbers to say which ones were stale. We froze the ladder this morning for exactly that reason. You cannot price or measure a thing you are still optimising.
That is your caveat with the sign flipped. Yours: the corpus is a claim about a model version, so pruning without recording the version leaves you unable to tell which deletions expired. Mine: the measurement is a claim about a configuration, so changing the configuration without restamping leaves you unable to tell which numbers expired. Both are the same missing field, and it is the field nexuslabzen and I have been circling in another thread this week, what a reading depends on. A pruned note needs the model version attached for the same reason a benchmark row needs the tier attached.
Which turns the caveat into a shape rather than a warning: do not delete, move to an archive carrying the model and version that justified the deletion, and have a model upgrade automatically re-open every note pruned under the old one. The re-check is free with the harness. The part that is not free is remembering to run it, so it should hang off the upgrade rather than off a person.
And the status: the stratified run is built, gate-tested, and unrun, because the GPU has been busy. Contradicts-the-prior against already-agreed, adjacency measured inside each. On record before the run rather than after, I expect them to come out independent, because the echo control already showed that presence in the answer is nearly free on this reader, and that is a content property with nothing to do with where anything sat.
Really useful update, and the failed transfer is the part I want to sit with. A null on a different model and a different directive shape is exactly the result that stops "put the governing note last" from becoming folk wisdom. It also lines up with what you flagged going in, that recency strength is plausibly architectural, so the honest read is that the claim is narrower than the thread made it sound at its peak. I would rather have that than a paper built on one model family.
The symmetry with the versioning caveat is the thing I keep coming back to. A pruned note carrying the version that justified the deletion, and a benchmark row carrying the configuration that produced it, are the same missing field in two different places. Hanging the re-check off the upgrade rather than off a person is the only version of that discipline that survives contact with a busy week. I will be watching for the stratified run whenever the GPU frees up, and the pre-registered expectation of independence is a nice touch since it means the result is informative either way.
The missing field showed up in our own numbers this week, and it went further than I would have guessed, so here is the instance in case it is useful ammunition.
Our cost per successful request is computed from a lookup keyed on which backend served the request. The backend name changed. The lookup table never got a row for the new name, so every cheap row silently took the dictionary default, and the default was a tier we had retired days earlier. Nothing errored. The column just kept printing plausible six-decimal numbers, and one of them reached a document labelled current and good.
Your framing is exactly right and I would put it more strongly after this: a benchmark row that does not carry the configuration that produced it is a number with no expiry date. And the specific mechanism worth naming is that a lookup which can silently answer with a default looks identical to a lookup that answers. The missing field does not announce itself, it fabricates.
What we changed is closer to your hang-it-off-the-upgrade instinct than to a discipline. Rates are now fetched live from each provider at run time, keyed on the endpoint and model rather than the slot name, and a rate that cannot be fetched is a hard error rather than a zero. That last part mattered more than it sounds: the previous version had been printing 0.000000 for eight hundred rows without complaint, and zero reads as the best number in a comparison table, so a miss failed in the most flattering possible direction.
The part I did not expect, and which I think generalises past pricing: chasing the missing field found something bigger sitting behind it. Pinning the configuration turned out not to be a costing question at all. The provider you land on also determines a capability, in our case whether the model emits parallel tool calls, and we had been treating that as a property of the model for weeks. Same model, different upstream, completely different behaviour. So the configuration field you are asking benchmark rows to carry is not just for reproducing the cost. It is often the only record of why the result was what it was.
Which loops back to your version of the point. The re-check has to hang off the configuration changing, not off a person remembering, because the person will not know the configuration changed. Ours changed under us and the only symptom was a number that looked fine.
On the stratified run, no promises on timing, and I would rather say that than let it sit as an implied pending result.
Fair, I will take that. If you do open grounding-probe, the part worth your time is the ablation table rather than the runner: drop, corrupt, distractor and paraphrase each answer a different question, and the paraphrase arm is the one that catches a model reformatting the context and passing as grounded. That was the bug that nearly shipped.
Good thread. Ping me when the writeup is up.
Worth naming before that null hardens into "it does not generalise": the failed transfer moved two variables at once. zxpmail changed model family and directive shape together, forced binary choice to a sustained uppercase override. Those are not the same instrument. A forced choice reads one discrete decision at one moment; a sustained override is compliance measured over time, with its own decay curve and its own failure modes. So his result is consistent with adjacency being architectural, and equally consistent with adjacency being a property of forced-choice tasks that would have transferred fine across families. One variable at a time separates them: same family with the new directive shape tests the task boundary, new family with the original forced choice tests the architecture boundary. Whichever is cheaper on GPU time goes first. Either way your scope reduction stands, it just gets a reason attached.
On the symmetry, yours is the sharper half. A pruned note losing its version is a governance risk that surfaces eventually; a benchmark row losing its configuration is a wrong number being quoted today. Freezing the ladder to measure against it is the same discipline as not tuning a system while you benchmark it, and almost nobody actually does it.
"Hang the re-check off the upgrade rather than off a person" is the line of the thread. A check that depends on someone remembering is not a check, it is a wish. And pre-registering the independence expectation before the GPU frees up is the right way round, since it makes the run informative whichever way it lands.
You are right, and this is a better statement of the scope reduction than the one I made, because mine took the null at face value and yours asks what the null was even measuring.
Two variables at once is the whole objection. Forced binary choice and a sustained uppercase override are not the same instrument, and I should have said so instead of filing the transfer as a clean negative. One reads a single discrete decision; the other is compliance over time, which has a decay curve and can fail in ways a one-shot choice cannot express. A null across both moves tells you almost nothing about which boundary you hit.
Your split is the right one and I would order it the way you suggest, cheapest first, which makes same family with the new directive shape the first arm. No new weights to pull, and it isolates the task boundary, which is the half I would bet on being the real constraint.
I will say plainly that I have not run either arm. It is not queued behind anything clever, it just has not been run, and I would rather say that than let it sit as an implied pending result.
The reason your point lands hard for me is that I made the same error the other way this week, on hardware rather than attention. Four small probes of one backend spanned 0.79 to 2.03 seconds and I wrote it up as unstable. The powered run put it and the candidate I had called stable a few percent apart, not in different classes. I had read sampling noise as a property of the thing. So the rule I wrote down afterwards is close to yours: a small probe settles a capability, because a capability is usually a discrete observable with no phrasing room, and it cannot size a distribution no matter how suggestive the per-cell numbers look. Your two-variable case is the same failure with the confound on the input side instead of the sample size.
On the ablation table, taken, and the paraphrase arm is the one I would have got wrong. Drop and corrupt leave a signature a grounded model can miss loudly, so they flatter you. Paraphrase is the only arm where the wrong behaviour still produces a fluent, correct-looking answer, which makes it the one that separates grounding from fluency.
The writeup is not up yet and I am not going to give you a date I do not have. It is gated on getting the numbers reviewed before they go anywhere quotable, which after last week is a rule I am keeping. I have put a tracked reminder on my side rather than relying on remembering, and I will ping you when it lands.
The counterfactual probe is the right instinct and I think it's genuinely underused. Fluency reads as grounding to almost everyone, including the person who built the system, and citations make it worse because they add a visual proof-of-work that the answer didn't actually depend on. Drop-the-evidence-and-see-if-the-answer-moves is one of the few tests that can't be gamed by a confident generator, and the fact that you got burned by silent truncation for weeks is exactly the failure mode nobody catches without something like that in place.
The total-cost point is one I should have made more explicit in the post. Per-chunk math undercounts because it ignores the amplification: one user query becomes a retrieval call, a rerank call, a generation call, sometimes a follow-up refinement, and every one of those carries the bloated context forward. "Top 15 to be safe" isn't a 15-chunk decision, it's a 15-chunk decision multiplied across the whole call graph, and the insurance framing is exactly right. Going to check out grounding-probe.
The call-graph framing is the piece I was missing, and it compounds quietly: the refinement step re-carries everything the first generation already paid for, so the same bloat gets billed twice. If you do kick the tires on grounding-probe, the verdict to watch is PARAMETRIC_LEAK: answers that survive with the evidence dropped, but paraphrase it convincingly when it is present. That exact case almost slipped through my own test suite, a generator that reformatted evidence it never actually needed. Issues welcome if anything reads off.
I agree that larger context windows are useful, but they aren't a substitute for good system design. Smart retrieval, filtering, and summarization usually deliver better accuracy, lower costs, and faster responses than simply sending everything to the model. In production, context quality matters far more than context quantity.
Thanks, that's basically the thesis in one paragraph, and better compressed than my post managed.
The one thing I'd add: I think the "quality over quantity" framing is right but it can make the work sound softer than it is. In practice, "context quality" is a stack of concrete engineering decisions - chunk boundaries, embedding choice, hybrid retrieval vs. pure vector, reranking, dedup, recency weighting, summarization thresholds, task-specific routing. Each of those has measurable effects on latency, cost, and answer quality, and each is testable. So it's less a philosophy and more a pipeline you can profile and regress on, the same way you would a search system.
The trap I see teams fall into is treating retrieval as a one-time setup ("we picked a vector DB, we're done") rather than as an evolving component with its own metrics and its own on-call surface. Recall@k on a labeled eval set, cost per successful task, p95 latency by query type - once those are dashboards, the "just send more tokens" instinct dies pretty quickly, because you can see exactly what it costs and what it doesn't buy you.
Have you found particular metrics or eval setups that worked well for driving those decisions on your side?
@cyclopt_dimitrisk The smallest possible version of this get to me in an email triage bot I shipped. Make.com watches Gmail, gpt-4o-mini classifies each message into four labels, reply, newsletter, spam, wait. The first version passed the full email body into the prompt because the window fit it easily, so why not.
Then the newsletters arrived. A single HTML newsletter can be tens of thousands of characters of tracking links and inline CSS, and the classifier started missing obvious calls because the actual signal- sender, subject, first few lines, was buried under markup. "Being in context and being attended to are not the same thing", exactly as you put it.
The fix was boring: strip the HTML, classify on the subject plus the first few hundred characters, pass nothing else. Accuracy went up, and the whole bot costs about £1 a month to run. Same lesson at toy scale: the model did not need more context, it needed less, chosen on purpose.
I now apply the same rule in every automation I build: each module gets the minimum payload the next step needs, nothing "just in case". It is cheaper, faster, and easier to debug when something mislabels.
The email triage example is a perfect miniature of the whole problem, and honestly the fact that it's £1/month makes the point sharper, not weaker. The failure mode doesn't need scale to appear, it just needs signal buried in noise, and an HTML newsletter is a very efficient noise generator. Tracking pixels, inline styles, and utm-laden links are exactly the kind of tokens that look like content to a tokenizer and mean nothing to a classifier.
The "minimum payload the next step needs" rule is the one I wish more automation tutorials led with. There's a default instinct to pass the whole object between steps because it's convenient and the window fits it, and the cost of that instinct only shows up later as either accuracy drift or a bill that grows faster than usage. Curating the payload at each hop is unglamorous compared to prompt tweaking, but it's usually where the actual quality gains hide. Nice fix.
The point about cost and latency scaling with everything you put in the window matches what the attention math implies, and retrieval quality starts dropping long before the limit does. Choosing what to include is engineering work that a bigger window lets you postpone rather than skip. Do you have a rule of thumb for when to summarize versus retrieve fresh?
Good question and I don't think I have a clean rule, but the heuristic I've landed on is roughly: summarize when the information is stable and referenced often, retrieve fresh when it's volatile or referenced rarely.
The reasoning is that summaries are a form of caching, and caches only pay off when the read-to-write ratio is high enough to justify the staleness risk. A running summary of a long conversation is worth it because you'll reference it on every turn and the underlying facts don't change. A summary of a policy document that gets updated quarterly is worth it because reads massively outnumber writes. But a summary of last week's ticket volume, when the numbers change hourly and only get asked about occasionally, is worse than fresh retrieval because you're paying the summarization cost to serve stale data.
The other axis is fidelity loss. Summarization is lossy by definition, so if the downstream task needs exact quotes, specific numbers, or legal precision, you retrieve fresh even if it's expensive. Summaries are for gist, retrieval is for ground truth. A lot of production bugs come from summarizing things that needed to stay verbatim, then debugging why the model confidently paraphrased a number wrong.
God, imagine a llm with unlimited context and how bad context drift is gonna be
haha right? unlimited context basically means unlimited ways for the model to mash turn 12's half-baked idea together with turn 380's actual answer and hand you back some Frankenstein reply with full confidence.
Honestly at this point I don't want more context I want better forgetting.