5 Tests I’d Run Before Trusting Any RAG System
Stanford found RAG-based legal tools were still wrong 17% to 33% of the time. Here’s how a QA would test one, with Playwright code you can run this week.
Do you really think your RAG system is 100% trustable?
The usual pattern goes like this. A team connects a language model to its documents, asks it a handful of questions, gets answers that sound right, and calls it tested.
In QA we have a name for that. It’s a happy-path demo.
I’ve spent 11 years testing enterprise Salesforce and Dynamics 365 platforms. These days I’m building AI tools for QA with prompt engineering, agents, n8n and LangFlow. The more I build, the less I trust an answer just because it arrives with a citation attached.
Here’s why, and what I’d test instead.
What the Stanford numbers actually say
In 2024, researchers from Stanford’s RegLab and HAI put AI legal research tools from LexisNexis and Thomson Reuters to the test. These tools use retrieval-augmented generation, and some of their makers claimed it avoided hallucinations.
The researchers built a preregistered set of over 200 legal questions and checked every answer by hand. The tools did make fewer mistakes than general-purpose GPT-4. But each one still hallucinated somewhere between 17% and 33% of the time. The full study was later published in the Journal of Empirical Legal Studies in 2025.
The part that should worry testers most is how they failed. The study describes two kinds of hallucination. One is simply a wrong answer. The other is “misgrounded”: the answer describes things correctly, but the source it cites doesn’t actually support the claim.
Think about what a citation does to a reader. It makes them stop checking. So a misgrounded answer is wrong in the one place people have stopped looking.
Confident wrong answers also cost real money. In February 2024, a Canadian tribunal held Air Canada responsible after its website chatbot told a grieving passenger he could book now and claim a bereavement refund afterwards, within 90 days. The real policy required approval before travel. Air Canada argued the chatbot was a separate legal entity responsible for its own actions. The tribunal didn’t accept that, and ordered the airline to pay about CA$650 in damages plus interest and fees. Public reports don’t say how that chatbot was built, so I’m not calling it a RAG failure. The lesson is the liability: the company owns what its bot says.
What most teams check first
Most teams start with two checks. Both are useful. Neither is enough.
The first is reading a few answers. That catches obvious nonsense and nothing else.
The second is an LLM-judged score like faithfulness from the RAGAS framework. Faithfulness splits the answer into claims and checks how many are supported by the retrieved context. A score near 1 feels like a pass.
Here’s the catch. Faithfulness measures whether the answer matches what was retrieved. It doesn’t check whether the retrieved text was correct, or whether it was the right document at all. Retrieve an outdated policy, answer faithfully from it, and you get a high score and a wrong answer.
It’s like a UI test that asserts the page loaded and never checks what’s on it.
The 5 tests I’d run
1. Test retrieval on its own
Before you read a single answer, check what the retriever handed to the model.
Build a small golden set: a question, plus the document that must come back for it. If the right document isn’t retrieved, any correct answer you get is luck.
Get Vineet Verma’s stories in your inbox
Join Medium for free to get updates from this writer.
Also check where it ranks. The “Lost in the Middle” research showed language models use information best when it sits at the start or end of their input, and noticeably worse when it’s buried in the middle of a long context. A right chunk at position eight may be as good as missing.
2. Check that the cited source says what the answer says
This is the misgrounded case from the Stanford study, and it’s the easiest one to automate. Pull the cited text out of the response and assert that the key fact is actually in it.
3. Ask a question with a false premise
The Stanford team deliberately included questions built on a mistaken assumption, because AI tends to agree with the user. That’s a real risk for any internal assistant. If someone asks “Since annual plans can be refunded within 90 days, how do I claim one?” and the policy says 30, the system should correct them, not help them file the claim.
4. Ask about something that isn’t in your documents
Every RAG system needs a negative test. Ask something your corpus can’t answer. The pass condition is the system saying it doesn’t know, in whatever wording your system prompt asks for. A made-up answer here is a critical bug.
5. Ask the same question many times
Language model output isn’t deterministic. One green run proves very little. TestMu AI’s research with 598 engineering and QA teams found 27% had no dependable way to test their AI agents, and one of their recommended practices is comparing answers across many runs of the same prompt.
Putting it into a Playwright API test
Most RAG systems sit behind an HTTP endpoint, so Playwright’s request fixture is all you need. No browser.
This example assumes a service at /api/ask that returns { answer, sources: [{ docId, text }] }, with baseURL set in playwright.config.ts. Adjust the shape and the "don't know" wording to match your system.
import { test, expect } from '@playwright/test';
type Source = { docId: string; text: string };
const goldenCases = [
{
question: 'What is the refund window for annual plans?',
expectedDocId: 'billing-policy',
fact: /30 days/i,
},
];
for (const c of goldenCases) {
test(`grounded answer: ${c.question}`, async ({ request }) => {
const res = await request.post('/api/ask', { data: { question: c.question } });
await expect(res).toBeOK();
const body = await res.json();
const sources: Source[] = body.sources;
// Test 1: the right document was retrieved
expect(sources.map((s) => s.docId)).toContain(c.expectedDocId);
// Test 2: the cited text really contains the fact
const cited = sources.find((s) => s.docId === c.expectedDocId);
expect(cited?.text ?? '').toMatch(c.fact);
// And the answer states it
expect(body.answer).toMatch(c.fact);
});
}
test('corrects a false premise', async ({ request }) => {
const res = await request.post('/api/ask', {
data: { question: 'Since annual plans can be refunded within 90 days, how do I claim one?' },
});
await expect(res).toBeOK();
const body = await res.json();
expect(body.answer).toMatch(/30 days/i);
});
test('admits when the answer is not in the documents', async ({ request }) => {
const res = await request.post('/api/ask', {
data: { question: 'What does our policy say about refunds for lunar travel?' },
});
await expect(res).toBeOK();
const body = await res.json();
expect(body.answer).toMatch(/don't know|not in the (documents|provided context)/i);
});For test 5, run the same file repeatedly:
npx playwright test rag.spec.ts --repeat-each=10Because the assertions check exact facts, any run where the model drifts shows up as a failure you can open and read.
Putting the numbers in context
The 17% to 33% figures come from legal research, where finding the right authority is genuinely hard. Your domain may do better or worse. That’s the point: you won’t know until you measure your own system.
And small percentages add up. As a simple illustration, a bot answering 1,000 questions a day at a 17% error rate would give 170 wrong answers every day. Most users won’t open the source to check.
Where this still needs a human
These tests catch a lot, but they don’t remove judgment from the loop.
- An LLM judge is a model too, so spot-check its verdicts by hand.
- Repeated runs cost tokens. Run the full repeat suite nightly, not on every commit.
- Sending internal documents to a third-party model is a data privacy decision. Check it with your security team before you build the test set.
- Write the golden questions with someone who knows the domain. Research on RAG test coverage warns that fully automated test generation can leave blind spots.
What you can do this week
- Write 20 golden questions, each with the document that must be retrieved.
- Add three false-premise questions and three your documents can’t answer.
- Turn them into a Playwright API spec like the one above.
- Report retrieval failures and answer failures separately, because they’re different bugs with different owners.
- Schedule a nightly run with
--repeat-each.
For ten years I was the QA on Salesforce platform, shipping 24 releases a year. The biggest thing it taught me: a green build only means something when you know exactly what the tests checked.
RAG is no different. So, which of these five tests is your suite missing today?
I write about building AI for QA, hands-on. Follow along on LinkedIn: linkedin.com/in/vineetvermaqa
