A medical RAG assistant that answered fluently, cited real sources, and passed every informal check was missing half of all emergency cases. Here is the architecture, the evaluation harness that caught it, the actual bug, and what it cost to fix.
The assistant looked finished.
It answered medical orientation questions in clear language, cited real passages from the corpus, refused things outside its scope, and handled every question we threw at it in review. We were discussing rollout.
Then the evaluation harness came online and reported that on emergency cases — the questions where being wrong is worst — recall was 50%.
Half. Not degraded, not marginal. A coin flip on exactly the category the system existed to get right.
This post is the whole story: what the system was, how it was built, what the bug actually was, how the evaluation found it, and what changed afterwards. It is the most useful thing I have learned about building AI for clinical settings, and none of it is about model quality.
TL;DR — key takeaways
- A system can be fluent, well-cited, and confidently wrong on the cases that matter. Aggregate quality tells you nothing about the dangerous stratum. - The bug was in routing, not retrieval or generation. Symptom text was being classified before it was normalised, so emergency red-flag phrases in casual phrasing never matched. - Stratified evaluation is what found it. Overall numbers looked healthy. Emergency-stratum recall was 50%, and only a per-stratum report made it visible. - Rule-based safety layers need evaluation too. Teams evaluate the LLM and treat the deterministic guard as obviously correct because it is simple. Simple code with wrong assumptions is still wrong. - Recall on the dangerous class is the metric. Precision matters, but for emergency detection a false positive costs a redirect and a false negative can cost far more. - Fixing it required no model change. Normalisation order, an expanded phrase set, and treating the classifier as fallible — then re-measuring.
What the system was
A medical-orientation assistant for a hospital. The job it did was narrower than "medical chatbot", and that narrowness was deliberate.
A user describes a problem in their own words. The system does three things: it retrieves relevant guidance from a curated corpus, it produces an advisory answer grounded in that guidance, and — before either of those matters — it decides whether the description contains signs that mean stop, this is not a question for an assistant, go to emergency services now.
The third job is the one that carries risk. The first two are a retrieval problem. The third is a safety problem, and it has a strong asymmetry: routing someone to emergency care unnecessarily is an inconvenience; failing to route someone who needed it is the failure mode that ends the project and hurts someone.
Everything about the architecture follows from that asymmetry.
The retrieval architecture
Retrieval is two-stage hybrid, which is now a fairly standard shape but worth spelling out because each piece is there for a reason.
Stage one: hybrid candidate generation. Dense retrieval over FAISS, plus BM25 lexical search, with the result sets merged.
Dense retrieval alone fails on exactly the queries clinical text produces: rare terms, drug names, specific anatomical vocabulary. Embedding models compress meaning, and low-frequency tokens are precisely what gets compressed away. BM25 has no such problem — it matches the literal string. Conversely BM25 fails when the user describes a symptom in lay language that shares no vocabulary with the clinical guidance.
Running both and merging covers the union of their strengths. This was not a marginal improvement in testing; it was the difference between usable and unusable on a meaningful slice of queries.
Stage two: cross-encoder reranking. The merged candidate set goes to a cross-encoder that scores each passage against the query jointly rather than comparing independent embeddings.
This is expensive, which is why it runs over a few dozen candidates rather than the whole corpus. It is also where most of the quality comes from. Bi-encoder retrieval gets the right passage into the top twenty reliably; the cross-encoder gets it into the top three. Given that the generation step only sees the top few passages, that reordering matters more than any prompt change.
Serving. The whole thing sits behind FastAPI, with a pytest suite covering the retrieval contracts and the safety behaviour.
The safety layer that failed
Ahead of retrieval sat a rule-based routing step.
The reasoning behind making it rule-based rather than model-based was sound and I would make the same call again. Emergency detection needs to be deterministic, auditable, and explicable to a clinician. "The model decided" is not an acceptable answer when someone asks why a case was not escalated. A rule you can read, test, and version is.
So: a set of red-flag patterns — symptom descriptions that indicate a potential emergency — matched against the incoming text. A match short-circuits everything else and returns an escalation message.
Deterministic. Auditable. Testable. And missing half the cases it was supposed to catch.
The evaluation harness
The reason we found out was a golden dataset of 1,000 questions with an independent LLM judge scoring faithfulness and hallucination.
Two design choices in that harness did the actual work.
Stratification. The dataset was not 1,000 questions drawn from one distribution. It was deliberately split into categories: routine lookups, ambiguous phrasings, multi-hop questions needing two sources, out-of-scope questions the system should refuse, and emergency cases.
Per-stratum reporting. Every metric was reported per category, never only as an aggregate.
That second choice is what surfaced the bug. The aggregate numbers were fine. Emergency cases were a modest fraction of the dataset, so their failure barely moved the overall figure. Had we reported a single blended score — as most projects do — we would have shipped.
The general principle, which I have written about at more length in how to evaluate a RAG system: an average over unequally important categories is not a summary, it is a disguise.
The actual bug
Debugging it came down to a single question: what exactly does the red-flag matcher see?
The red-flag matcher ran on the raw user text.
The phrase list had been written in clinical vocabulary — the terms that appear in the source guidance. Real users describing an emergency do not use those terms. They use casual, compressed, sometimes misspelled language, with different word order, extra filler, and lay synonyms.
There was a normalisation step in the pipeline — lowercasing, whitespace and punctuation handling, some term mapping. It ran after the routing check, because routing was placed first in the pipeline on the entirely reasonable grounds that safety should come before everything else.
Safety-first ordering, applied to a component that needed normalised input, produced a component that saw exactly the input it could not handle.
Emergency phrasings that matched the clinical vocabulary were caught. Everything else — which was roughly half the emergency stratum — went straight through to ordinary retrieval and got a calm, well-cited, thoroughly inappropriate informational answer.
Two things about this bug are worth sitting with.
It was not subtle in hindsight. It was a fifteen-minute fix once identified.
And it was invisible to every form of testing we had done except the stratified evaluation. Manual review missed it because the people testing were writing questions in the vocabulary they had absorbed from the source documents. The unit tests missed it because they were written from the same phrase list the matcher used — the tests and the code shared the wrong assumption. Aggregate metrics missed it by construction.
Why "it's just a rule" is a trap
The failure has a general shape worth naming, because it will recur in other systems.
When a pipeline mixes probabilistic components and deterministic ones, evaluation attention flows to the probabilistic parts. The LLM is the thing that might hallucinate. The retriever is the thing that might miss. Those get harnesses, metrics and dashboards.
The rule-based guard gets a code review, because it is simple, and simple things are assumed correct.
But simplicity guarantees only that the code does what it says. It says nothing about whether what it says is right. A regex is perfectly deterministic about matching the wrong thing. A rule written against clinical vocabulary is perfectly reliable at not matching lay vocabulary.
Deterministic components need behavioural evaluation exactly as much as models do. Not more tests of the implementation — tests of the assumption, against real input distribution.
The fix
Three changes, none of them to the model.
Normalise before routing. The obvious one. Text normalisation moved ahead of the red-flag check, so the matcher saw the same canonical form as everything downstream.
Expand the phrase set from real language. Rewriting the red-flag patterns using lay phrasings, common misspellings, and the multiple ways a person might describe the same symptom under stress — including in the languages users actually write in. This is a domain-knowledge task, not an engineering one, and it needs clinical input.
Treat the classifier as fallible. Instead of routing being a single binary gate, ambiguous cases bias toward escalation. Given the cost asymmetry, a system that over-escalates at the margin is behaving correctly.
After the fix, emergency-case recall on the golden dataset went from 50% to 100%.
I want to be precise about that number, because it is the kind of figure that invites overclaiming. It is recall on the emergency stratum of a 1,000-question evaluation dataset, after fixing a bug that dataset revealed. It is not a claim about all possible emergency phrasings in the world. It is a claim that the specific failure mode was closed and verified against the instrument that found it — and that the instrument now runs continuously, so a regression would surface.
What we changed structurally
The bug was cheap to fix. The process changes it prompted were the actual output.
Every deterministic safety component gets an evaluation stratum. If a component exists to prevent a bad outcome, there is a category in the golden dataset that measures whether it does, on realistic input.
Test data is written by people who did not write the code. The unit tests failed because they inherited the implementation's assumptions. Adversarial phrasings now come from outside the pipeline's vocabulary.
Per-stratum thresholds in CI. Emergency recall has a hard floor. A change that drops it fails the build. Aggregate scores are not permitted to be the gate.
Prompt-injection filtering ahead of every LLM call. Related hardening from the same review: a medical assistant ingesting retrieved documents is ingesting text it did not author, and that text can carry instructions. Filtering runs before the model sees anything, not as a post-hoc output check.
An explicitly advisory workflow. The assistant defers to physicians and emergency services by construction. It is not a diagnostic tool and its outputs do not read as diagnoses. That constraint is easier to hold when it is designed in rather than added as a disclaimer.
What generalises
Not much of this is specific to medicine. The transferable parts:
Find the stratum where being wrong is worst, and measure that separately, always. Every system has one. In a support bot it is billing disputes. In a code agent it is destructive operations. In a hiring tool it is the protected-class slice. Whatever it is, its performance is not visible in an aggregate.
Order of operations in a pipeline is a correctness property. Placing safety first is a good instinct that produced a bug here, because the component's input requirements were not considered alongside its priority. Ask what each stage needs to see, not only when it should run.
Your tests and your code should not share a source of truth. If the phrase list drives both the matcher and its tests, the tests can only verify the list was applied, never that the list was right.
Rule-based does not mean verified. It means auditable. Those are different, and conflating them is how a deterministic component becomes the least-examined part of a system.
Closing
The uncomfortable thing about this bug is how well the system performed while it had it.
It was fluent. It cited real sources. It refused out-of-scope questions correctly. Every human who tested it came away impressed, including me. There was no visible symptom, because the failure mode was producing a good answer to the wrong question — and a good answer looks like success.
The only reason it was found before deployment is that somebody built an instrument capable of measuring the specific thing that was broken, and reported it separately instead of averaging it away.
That is the entire lesson. Not "test your code" — we tested it. The lesson is that the thing you most need to measure is usually a small, awkward, expensive-to-construct subset that your aggregate metrics are actively hiding from you, and you have to go looking for it on purpose.
If you are building AI for a clinical or otherwise high-stakes setting and want a second pair of eyes on an evaluation design, get in touch.
Frequently asked questions
What is a clinical RAG assistant?
A retrieval-augmented generation system that answers medical questions by retrieving passages from a curated clinical corpus and generating an answer grounded in them, rather than from the language model's parametric memory. In a safety-conscious design it is explicitly advisory: it orients the user, defers to clinicians, and escalates anything showing emergency red flags rather than attempting to handle it.
Why use hybrid retrieval instead of vector search alone in medical RAG?
Because dense embeddings systematically lose low-frequency terms — drug names, rare conditions, specific anatomical vocabulary — which is exactly the vocabulary clinical queries contain. BM25 matches those literally. Conversely, BM25 fails when a user describes a symptom in lay language sharing no words with the clinical text, where dense retrieval succeeds. Running FAISS and BM25 together and merging the results covers the union of both strengths, and in this system it was the difference between usable and unusable on a meaningful slice of queries.
What does a cross-encoder reranker actually add to a RAG pipeline?
It scores each candidate passage jointly with the query rather than comparing independently computed embeddings, which is substantially more accurate but too expensive to run over a whole corpus. The practical effect: bi-encoder retrieval reliably gets the right passage into the top twenty, and the cross-encoder gets it into the top three. Since generation only sees the top few passages, that reordering typically improves answer quality more than prompt engineering does.
How do you measure whether a medical AI assistant is safe?
Not with an aggregate score. Build a stratified evaluation set where safety-critical cases are their own category, and report recall on that category separately and continuously. In this system the overall metrics looked healthy while emergency-case recall sat at 50%, because emergency cases were a small fraction of the dataset. The safety metric has to be a hard threshold in CI, not a line item in an average.
Why did a rule-based safety check fail when the language model worked?
Because the rule ran before text normalisation and its phrase list was written in clinical vocabulary, while real users describe emergencies in casual, compressed, sometimes misspelled lay language. The rule was perfectly deterministic — it just deterministically matched the wrong thing. Rule-based components are auditable, which teams frequently mistake for verified; they need behavioural evaluation against real input distributions exactly as much as models do.
What is emergency red-flag detection in a medical chatbot?
A gate that scans a user's description for signs indicating a potential medical emergency and, on a match, stops the normal answering flow and directs the user to emergency services. It carries a strong cost asymmetry — over-escalating is an inconvenience, under-escalating can be severe — so a correctly tuned system biases toward escalation on ambiguous input rather than optimising for precision.
How do you protect a medical RAG system from prompt injection?
Filter before the model, not after. A RAG system ingests retrieved documents it did not author, and that text can carry instructions aimed at the language model. Prompt-injection filtering runs ahead of every LLM call rather than as an output check, combined with deduplication, version-safe index rebuilding, and a strict advisory-only workflow that constrains what the system is able to assert in the first place.
Related reading
- Why Medical AI Fails in Production — and Why Data Quality Decides It — the data-side failures that surround this one. - How to Evaluate a RAG System — And the Bug My Evaluation Caught — the general evaluation method, in full. - AI Agents from Prototype to Production: What Actually Breaks — the same class of problem in agent systems.