Member-only story
Did IBM Just Kill RAG? The Answer Will Surprise You
The viral claim doesn’t hold up. The problem behind it does, and it’s probably in your pipeline right now.
RAG is officially cooked.
That’s what a post in my feed said this week. IBM, it claimed, had built a retriever that hallucinates 65 x less than fine-tuned RAG.
I found the paper. It’s from IBM, and the 65x is real: their retriever, STAIR, pointed to a book section that doesn’t exist 0.05% of the time, against 3.25% for another fine-tuned retriever.
But that’s what “hallucination” means there, rather than a wrong answer. And STAIR is a retriever built for RAG.
The problem the post describes is real, though. In a 2024 Stanford study, a RAG-based legal research tool was asked about opinions written by Judge Luther A. Wilgarten.
The judge never existed.
The tool answered anyway, with a real case and a real citation.
A wrong answer with a real citation is harder to catch than a wrong answer with none.
Your RAG system can make the same mistake.
Fixing it doesn’t mean throwing RAG away. It means checking what the retriever hands the model, and below are five…
