Most RAG systems are tested by asking them a few questions and reading the answers. That is not evaluation. Here is how to build a golden dataset, use an LLM as a judge without fooling yourself, and why aggregate accuracy hides the failures that matter most.
Building a RAG demo takes an afternoon. Chunk some documents, embed them, drop them in a vector store, put an LLM in front, ask it three questions, watch it answer correctly, ship it.
Building a RAG system you would let someone rely on takes considerably longer. Almost all of that extra time goes into one thing: finding out where it fails.
I learned this the expensive way while building a medical-orientation assistant. The system looked good. It answered questions fluently, cited real sources, and passed every informal check we threw at it. Then we built a proper evaluation harness, and it turned out that on the single most dangerous category of question, it was wrong half the time.
This post is about how to find that kind of problem before your users do.
TL;DR — key takeaways
- Reading a few answers is not evaluation. You only test what you thought of, fluent and correct look identical, and you cannot detect regressions. - Measure retrieval and generation separately. If you only score the final answer, you cannot tell whether to fix the retriever or the prompt — and you will usually guess wrong. - Retrieval recall sets your ceiling. If Recall@10 is 0.6, no prompt can make the system right more than 60% of the time. Fix retrieval first; it is also the cheapest thing to measure. - Build a golden dataset stratified by question type, not just by volume. Include refusals, adversarial phrasings, and the high-stakes cases. - Calibrate your LLM judge before trusting it. Score it against hand-labelled examples; an uncalibrated judge produces confident, meaningless numbers. - Report per stratum, never only the aggregate. A 94% average hid a category where our system was wrong half the time — and it was the category where being wrong mattered most.
"It looks right" is not evaluation
The default way people test a RAG system is to type questions into it and read the answers.
This fails for three reasons.
You test what you think of. You ask the questions you had in mind when you designed the system. Those are exactly the questions it handles well. Your users will ask different ones.
Fluent and correct look identical. An LLM producing a confident, well-structured, completely fabricated answer looks the same as one producing a confident, well-structured, correct answer. You cannot tell them apart by reading unless you already know the answer.
You cannot detect regressions. You change the chunk size, swap the embedding model, adjust a prompt. Did it get better? You have no idea. You ask your three questions again, they still look fine, you ship.
Manual checking is useful for catching catastrophic failures. It is useless for catching the ones that matter.
Evaluate retrieval and generation separately
This is the single most important structural decision, and most teams skip it.
A RAG system has two failure modes that need completely different fixes:
1. Retrieval failed. The right information was never pulled from the corpus. The LLM had no chance. 2. Generation failed. The right information was retrieved, and the LLM still produced a wrong answer — ignored it, misread it, or invented something on top of it.
If you only measure the final answer, these collapse into one number and you cannot tell which one to fix. You will spend a week tuning prompts when your retriever was the problem all along.
Measure them separately:
| Stage | What you are asking | Metrics | |---|---|---| | Retrieval | Did the right chunk make it into the context? | Recall@k, MRR, nDCG, hit rate | | Reranking | Is the right chunk near the top? | Recall@1 vs Recall@10 gap | | Generation | Given the context, is the answer grounded in it? | Faithfulness, hallucination rate | | Generation | Does the answer address the question? | Answer relevance | | End-to-end | Would a domain expert accept this? | Correctness, human review |
The retrieval metrics are the cheap ones. They need no LLM, they run in seconds, and they tell you your ceiling. If Recall@10 is 0.6, your system cannot possibly be right more than 60% of the time, no matter how good your prompt is. Fix retrieval first.
The gap between Recall@1 and Recall@10 is worth watching on its own. If the right chunk is usually somewhere in the top ten but rarely first, you do not have a retrieval problem — you have a ranking problem, and a cross-encoder reranker will fix it far more cheaply than a better embedding model.
Build a golden dataset
A golden dataset is a set of questions paired with the answer you expect and, ideally, the source chunk that should have been retrieved.
For the medical assistant, we built 1,000 of them. That number matters less than the structure. A thousand questions all drawn from the same easy distribution is worse than two hundred that deliberately span the range.
What makes a golden dataset good:
Stratify by category, not just volume. Split the dataset into the kinds of question your system will actually face — routine lookups, ambiguous phrasings, multi-hop questions requiring two sources, out-of-scope questions it should refuse, and the high-stakes cases where being wrong is expensive. Track metrics per stratum. More on this below, because it is where the real bug was hiding.
Include questions your system should refuse. A RAG system that answers everything is broken. If a question falls outside the corpus, the correct behaviour is to say so. If you have no refusal cases in your dataset, you are implicitly rewarding the model for guessing.
Include adversarial phrasings. Same underlying question, phrased casually, phrased with typos, phrased in another language, phrased with the key term missing. Retrieval is far more brittle to phrasing than people expect.
Write down the expected source, not just the expected answer. This is what makes retrieval measurable independently. It is tedious. Do it anyway.
Do not generate the whole thing with an LLM. Synthetic questions are useful for volume, but an LLM generating questions from your corpus produces questions that are trivially answerable from your corpus. They inflate your scores and teach you nothing. Use synthetic data to pad the easy strata; write the hard cases by hand, ideally with a domain expert.
Use an LLM as a judge — carefully
Once you are past retrieval, you need to score generated answers, and you cannot do that with string matching. Real answers are correct in many different wordings.
So you use a second LLM as a judge. This works well, with caveats.
Use a different model than the one being evaluated. A model judging its own output is measurably biased toward it. Independence is the entire point.
Judge one dimension at a time. Do not ask "is this answer good, score 1-10." That number means nothing and is not reproducible. Ask narrow, binary-ish questions with an explicit rubric:
- Faithfulness: is every claim in this answer supported by the provided context? Yes / No / Partially. - Relevance: does this answer address the question that was asked? - Refusal correctness: the system declined to answer — was declining the right call?
Give the judge the context, not just the answer. Faithfulness is a relationship between the answer and the retrieved chunks. A judge that only sees the answer is grading on plausibility, which is exactly the thing you are trying to detect.
Calibrate the judge against humans. Label 50-100 examples by hand, run the judge on the same examples, and measure agreement. If the judge disagrees with you 30% of the time, your evaluation numbers are noise and you need to fix the rubric before trusting anything downstream. This step gets skipped constantly and it invalidates everything built on top of it.
Here is roughly the shape of the harness:
The important detail is the last line. Not aggregate(results) — aggregatebystratum(results).
The bug: aggregate metrics hide the failures that matter
Here is what happened on the medical assistant.
The system had a routing layer in front of retrieval. Most questions went through the normal RAG path. But some questions — ones containing emergency red flags, the symptoms where the correct response is stop reading this and call emergency services — were supposed to bypass RAG entirely and return a fixed, safe response.
Overall, the evaluation looked healthy. Faithfulness was good. Retrieval recall was good. Refusal behaviour was good. If we had looked at a single headline number, we would have shipped it.
But the golden dataset was stratified, and the emergency stratum was reported separately. On that stratum, recall was 50%.
Half of the emergency-flagged questions were not being routed to the emergency path. They fell through to normal RAG, which answered them helpfully and calmly — retrieving genuinely relevant medical information, staying perfectly faithful to it, and scoring well on every generation metric. The answers were good. They were also the worst possible response to that question.
That is the part worth sitting with. Every generation metric was fine. Faithfulness was fine, because the answer was faithful to the retrieved context. Relevance was fine, because the answer did address the question. The system was not hallucinating. It was doing exactly what it was built to do, on a question where doing that was wrong.
No amount of prompt tuning would have surfaced this. It was a routing bug, and only a stratified evaluation with a category specifically for dangerous inputs could expose it. After fixing the routing logic, emergency recall went to 100%, and it stayed there because the harness now runs on every change.
The general lesson: your aggregate metric is dominated by your most common case, and your most common case is usually your easiest one. A system at 95% overall can be at 50% on the 3% of traffic where errors are unacceptable. If you do not slice your metrics by category, you will never see it.
Pick the failure mode you can live with
Most evaluation advice treats precision and recall as a technical trade-off to optimise. In practice it is a product decision, and someone has to make it explicitly.
For a screening system, false negatives are worse than false positives. Missing something real is a serious failure; flagging something that turns out to be nothing costs a review. So you tune toward recall and accept the extra false positives.
For an assistant that answers factual questions, it usually inverts. A confident wrong answer damages trust more than a refusal does. There, you tune toward precision and accept that the system says "I don't know" more often than strictly necessary.
Decide which one you are before you look at the numbers. Otherwise you will rationalise whatever the model happens to do.
Make it a regression test
An evaluation you run once is a report. An evaluation you run on every change is a safety net.
Wire the golden dataset into your test suite. Set thresholds per stratum — not one global threshold — and fail the build when a stratum regresses, even if the overall average improved. That asymmetry is the whole point: an average that improves while your emergency stratum degrades is not an improvement.
Keep the harness fast enough that people actually run it. Retrieval metrics need no LLM at all and can run on every commit in seconds. Reserve the expensive LLM-judged pass for pull requests or nightly runs.
And re-run it after changes that feel unrelated to quality. Rebuilding an index, upgrading a dependency, changing a chunking parameter — these silently move retrieval behaviour more often than you would expect.
What I would tell someone starting today
Before you tune a single prompt:
1. Write 100 questions by hand, spanning easy, ambiguous, out-of-scope, and dangerous. Record the expected source for each. 2. Measure retrieval recall alone. Fix retrieval until the ceiling is high enough to be worth pursuing. 3. Add an independent LLM judge for faithfulness and relevance. Calibrate it against 50 hand-labelled examples before trusting it. 4. Report every metric per stratum. Never look at the aggregate on its own. 5. Put thresholds in CI and let them fail the build.
None of this is glamorous, and none of it will show up in a demo. It is the difference between a system that looks like it works and one that you know the failure modes of.
The question to ask about a RAG system is not:
"Does it answer correctly?"
It is:
"On which questions is it wrong, and can I live with those?"
You cannot answer that by reading a few outputs. You have to measure it.
Frequently asked questions
What metrics should you use to evaluate a RAG system?
Split them by stage. For retrieval: Recall@k, MRR and nDCG, which need no LLM and run in seconds. For generation: faithfulness (is the answer grounded in the retrieved context) and answer relevance (does it address the question). End-to-end: correctness as judged by a domain expert. Reporting a single blended score is the most common mistake, because it makes the two very different failure modes indistinguishable.
How big should a golden dataset for RAG evaluation be?
Structure matters more than size. Two hundred questions that deliberately span easy lookups, ambiguous phrasings, multi-hop questions, out-of-scope refusals and high-stakes cases will teach you far more than two thousand drawn from the same easy distribution. We built 1,000 for a medical assistant, but the value came from stratification, not the count. Start with 100 written by hand and grow from there.
What is LLM-as-a-judge and when should you not trust it?
LLM-as-a-judge uses a separate language model to score generated answers on criteria like faithfulness and relevance, because string matching cannot handle answers that are correct in many wordings. Do not trust it until you have calibrated it: score 50 hand-labelled examples and check how often the judge agrees with you. Also use a different model from the one being evaluated, and give it the retrieved context so it scores grounding rather than plausibility.
Why does aggregate accuracy hide RAG failures?
Because your question categories are not equally represented or equally important. A system that is 94% accurate overall can be 50% accurate on the small category where errors are dangerous, and the aggregate will never show it. This is exactly what happened in the clinical assistant I built: emergency-case recall was 50% while every headline number looked healthy.
How do you detect hallucination in a RAG system?
Score faithfulness: given the retrieved context and the generated answer, ask whether every claim in the answer is supported by the context. Claims that are not supported are hallucinations regardless of whether they happen to be true. This is more useful than checking factual correctness alone, because a RAG system that produces true statements it did not retrieve is still broken — it means the model is answering from parametric memory rather than your corpus.
Should RAG evaluation run in CI?
Yes, with thresholds that fail the build. Retrieval metrics are fast and deterministic enough to run on every commit. Judge-based generation metrics are slower and cost money, so run them on a fixed subset per commit and the full set nightly. Without CI thresholds, evaluation becomes something you do once before launch and never again, which is precisely when regressions slip in.
Related reading
- From 50% to 100% Emergency Recall: Debugging Safety Routing in a Clinical RAG Assistant — the full story of the bug this evaluation caught. - Why Medical AI Fails in Production — and Why Data Quality Decides It — what happens after evaluation, when the model meets real clinical data. - RAG vs Fine-Tuning vs Prompt Engineering: How to Actually Choose — deciding whether you should be building a RAG system in the first place.