I built a RAG system. Here’s the bug I almost blamed on the wrong thing.
What a single wrong answer taught me about debugging retrieval-augmented generation, and about the evaluation metrics we use to measure it.
A few weeks ago, I asked my RAG system a simple factual question: “How many warehouses did Costco operate at the end of fiscal 2022, and how has that count grown over the prior two years?”
The system got it wrong. My first instinct was to blame the retrieval algorithm. Obviously it just didn’t find the right document. I was about to write “improve retrieval” in my notes and move on to the next thing.
I didn’t move on. Tracing that one wrong answer through the pipeline, layer by layer, taught me more about how RAG systems fail than building the system in the first place did.
The setup, briefly
I built retail-rag-eval-lab, a retrieval-augmented generation system that answers natural-language questions about retail companies (Walmart, Target, Costco, Kroger) grounded in about 90 real SEC filings pulled directly from EDGAR.
The pipeline follows the usual shape: parse the filings, chunk them into roughly 500-token pieces, embed the chunks, store them in a vector database, retrieve the most relevant ones for a given question, and have an LLM generate an answer grounded in what came back.
I also built a second retrieval path, a hybrid approach combining semantic embedding search with BM25 keyword search, fused together using a technique called Reciprocal Rank Fusion, and compared it against the plain semantic-only baseline. Hybrid search won on average. Hit rate went from 70% to 80%, and precision improved too.
But averages hide stories. The Costco warehouse question was one of the ones hybrid search still got wrong.
Resisting the easy explanation
It would have been simple to chalk this up as “retrieval failed on this one” and move to the next fix. Instead I asked a more useful question: at which layer did this actually fail? A RAG pipeline has at least three places a wrong answer can originate, and each one calls for a different fix.
I started with chunking, since blaming retrieval only makes sense if the fact was chunked correctly to begin with. I searched the processed data for the accession number of Costco’s real FY2022 10-K and found this sitting inside a single, intact chunk:
“Costco operated 838, 815, and 795 warehouses worldwide at August 28, 2022, August 29, 2021, and August 30, 2020, respectively.”
Complete, not split across a boundary. So the chunking wasn’t the issue.
Get sonam choudhary’s stories in your inbox
Join Medium for free to get updates from this writer.
Next, retrieval. The fact existed somewhere in my vector store, but did my hybrid retriever actually surface it? I ran the query and printed the rank of every result:
results = hybrid_retrieve(question, top_k=15)
for i, r in enumerate(results):
print(i+1, r['metadata']['source_path'])The correct chunk showed up at rank 11. My pipeline only passes the top 5 chunks to the language model, so it missed the cutoff by six places. Looking at ranks 1 through 10, the pattern was obvious once I saw it: nearly all of them were Costco filings from other fiscal years, 2024, 2025, 2021, each with almost identical “warehouse count” language sitting in a similar spot in the document. My retrieval system wasn’t confused about the company at all. It was confused about the year, because Costco’s 10-Ks repeat this phrasing nearly verbatim every year with different numbers attached. This was the real bottleneck.
Then generation, which is where things got more interesting than I expected. Given only wrong-year chunks, since the correct one never made the top 5, what did the model actually do? I assumed either a hallucinated number or a flat refusal. It did neither. It answered:
“Based on the context provided, I can give you the following warehouse counts from Costco’s 10-Q filings: May 8, 2022: 829 warehouses… February 13, 2022: 828 warehouses… However, the context does not contain enough information to answer the specific fiscal-year-end figure you asked about.”
Every number in that response was real, every citation accurate, and the model correctly flagged that none of it actually answered the question asked. Of the three layers, generation was the one behaving best.
My evaluation metric couldn’t tell the difference
I was scoring all of this with RAGAS, a standard RAG evaluation framework, and one of its metrics, answer relevancy, gave that response a flat 0.00. As if the model had said nothing useful whatsoever.
Looking across all ten of my test questions, relevancy scores turned out to be almost perfectly bimodal: either around 0.95 for a direct, confident answer, or exactly 0.00 for any hedge or admission of uncertainty. Almost no middle ground existed between the two.
That’s a real limitation of the metric, not a one-off fluke. Answer relevancy, as RAGAS computes it, generates hypothetical questions that a given answer would suit, then checks how closely those match the original question. An honest, partially useful, correctly caveated answer doesn’t reverse-engineer well into “what question was this answering,” so it ends up scoring the same as an answer that’s simply wrong.
Which means my hybrid retrieval system, despite finding the right documents more often (context precision nearly doubled, from 0.25 to 0.45), actually produced more of these low-relevancy refusals than the naive baseline. Not because it performed worse, but because it surfaced more borderline-relevant content that the model correctly judged insufficient, rather than clearly wrong content it might have answered from anyway. A dashboard showing “answer relevancy: down” would have told a misleading story on its own. Only tracing one real example all the way through revealed what was actually happening.
What actually needs to change
The fix here isn’t a smarter ranking algorithm. The real problem is that several years of near-identical language are competing for the same slots in my results, and no amount of clever fusion reliably wins that fight every time.
A more targeted fix: extract the filing year directly from each document’s SEC accession number, since it’s already embedded in the ID (0000909832–22–000021 was filed in 2022), attach it as metadata to every chunk, and when a question references a specific year, filter the candidate pool to that year before ranking anything at all. Remove the competition instead of trying to out-rank it.
That’s a specific, evidence-backed fix rather than a vague “improve retrieval” note, and I only landed on it because I refused to accept “retrieval failed” as a full explanation.
The takeaway
It’s tempting to trust an aggregate metric and move on, especially under time pressure. But the metric telling me hybrid search had regressed on relevancy was measuring something subtly different from what I actually cared about. The only way to find that out was to trace one real example through the whole system and read the actual output instead of just the score.
If you’re building or evaluating a RAG system, resist the urge to guess which component is at fault when something looks wrong. Check the data. Check the retrieval ranks. Read the generated text itself. The bug is rarely where your first instinct points, and the fix tends to be sharper once you’ve actually found it.
Full code, evaluation results, and write-up available on GitHub, retail-rag-eval-lab.
