𝗥𝗘𝗔𝗦𝗢𝗡𝗘𝗫 · 𝗥𝗘𝗖𝗔𝗟𝗟 𝗢𝗥 𝗥𝗘𝗖𝗢𝗡𝗦𝗧𝗥𝗨𝗖𝗧𝗜𝗢𝗡?
Our last experiment showed something uncomfortable:
More context didn’t mean better accuracy. Sometimes it got worse.
So we asked a sharper question:
If the same evidence is presented in a different order – or shortened all the way down to a small fragment – does the answer change?
And if it does, what’s actually changing?
It’s the question every RAG pipeline hits in production: retrieval returns the right documents, the model gives the right answer – but was it reasoning over what you retrieved, or pattern-matching to something adjacent it already knew?
An LLM can be right two ways:
- 𝗥𝗘𝗖𝗔𝗟𝗟 – surfaces an answer tied to something it’s seen before.
- * 𝗥𝗘𝗖𝗢𝗡𝗦𝗧𝗥𝗨𝗖𝗧𝗜𝗢𝗡 – rebuilds the answer from structure still present in the evidence.
At full context, these look identical.
So we built a controlled benchmark:
- 24 deterministic problems
- • SERIAL vs SET – cause→effect chains vs. the same entities with the dependency removed.
- • O1 – O4 – four orderings of the same evidence.
- • 100% → 5% retention
- • Randomized twins – swapped identifiers.
- 480 controlled cells per run.
𝗪𝗵𝗮𝘁 𝘄𝗲’𝗿𝗲 𝘀𝗲𝗲𝗶𝗻𝗴
The first 4 problems already show a pattern.
𝗦𝗲𝗾𝘂𝗲𝗻𝗰𝗲 𝗰𝗮𝗻 𝗱𝗼𝗺𝗶𝗻𝗮𝘁𝗲 𝗾𝘂𝗮𝗻𝘁𝗶𝘁𝘆
At the same retention level, accuracy ranged 0% → 100% by ordering alone.
𝗖𝗼𝗺𝗽𝗿𝗲𝘀𝘀𝗶𝗼𝗻 𝗰𝗮𝗻 𝗳𝗶𝘅 𝘁𝗵𝗲 𝗮𝗻𝘀𝘄𝗲𝗿
Two O2 cases were wrong at full context, then correct after compression.
Get Wagied Davids’s stories in your inbox
Join Medium for free to get updates from this writer.
More information wasn’t better – it was more distracting.
𝗘𝗿𝗿𝗼𝗿𝘀 𝗯𝗲𝗵𝗮𝘃𝗲 𝗹𝗶𝗸𝗲 𝗮𝘁𝘁𝗿𝗮𝗰𝘁𝗼𝗿𝘀
Compressed failures didn’t scatter.
They repeatedly landed on one alternative CVE, a real node still in the original chain.
O1 and O4 converged on the same attractor; O3 selected a different alternative (adjacent in pair 4).
𝗦𝗘𝗥𝗜𝗔𝗟 𝗮𝗻𝗱 𝗦𝗘𝗧 𝗱𝗶𝘃𝗲𝗿𝗴𝗲
SERIAL showed path-dependent answer selection, consistent with the recall hypothesis.
SET showed more evidence-dependent behaviour – but 42% of compressed SET cells (through problem 3) exhausted the token budget without an answer.
𝗪𝗵𝗮𝘁 𝘁𝗵𝗲 𝗲𝘃𝗮𝗹 𝘁𝗮𝘂𝗴𝗵𝘁 𝘂𝘀
The eval almost lied to us.
What looked like reasoning failure was partly a measurement artifact – the model’s own thinking consumed the output budget.
ReasonEx now scores:
𝗖𝗢𝗥𝗥𝗘𝗖𝗧 · 𝗪𝗥𝗢𝗡𝗚 · 𝗧𝗥𝗨𝗡𝗖𝗔𝗧𝗘𝗗
Audit the instrument before diagnosing the model.
𝗧𝗵𝗲 𝗵𝗼𝗻𝗲𝘀𝘁 𝗹𝗶𝗺𝗶𝘁
One model. One frozen benchmark. One run. 4/24 problems.
The randomized twin and model-scale ladder are next.
Because the real question isn’t:
“Can the model get the answer?”
It’s:
“When the evidence changes, is the model reconstructing the answer – or recalling
a competing one?”
That’s the RECALL ↔ RECONSTRUCTION boundary we’re chasing.
