The three ways to make a language model do what you want are constantly compared as if they were competitors. They solve different problems. Here is the decision rule, what each one actually costs, why most teams reach for fine-tuning when they needed retrieval, and how to tell which situation you are in.
This is the most common question I get asked by students and by teams starting their first serious LLM project, and it is almost always asked in the wrong form.
"Should we use RAG or fine-tune?"
The question assumes the two are alternatives competing to solve one problem. They are not. They fix different failures, and the reason so many projects waste months is that somebody picked the expensive one to solve a problem the cheap one would have solved in an afternoon.
There is a decision rule, it is short, and it works most of the time. This post is that rule, the reasoning behind it, what each option actually costs, and the cases where the rule breaks down.
TL;DR — key takeaways
- The rule: if the model lacks knowledge, retrieve. If it lacks behaviour, fine-tune. If it lacks instructions, prompt. Most projects need the first and reach for the second. - Start with prompting, always. It is the cheapest to try and the fastest to reverse, and it resolves more problems than people expect. - RAG fixes knowledge. Facts the model never saw, facts that change, facts that are private to you, and anything you need to cite. - Fine-tuning fixes behaviour. Format, tone, structure, a domain's idiom, and reliably following a task shape. It is a poor and expensive way to teach facts. - Fine-tuning does not reliably add knowledge. It shifts probabilities over how the model expresses itself far more than what it knows, and what it does absorb it cannot cite. - They compose. Retrieval for knowledge and a fine-tune for format is a common and correct production configuration. - Diagnose before choosing. Find out whether failures come from missing information or from wrong shape — the answer determines the tool.
The three things you can change
A language model produces an output from three inputs: the weights it was trained with, the context you give it, and the instructions in that context. Each technique modifies exactly one.
| Technique | What it changes | What it fixes | What it cannot fix | |---|---|---|---| | Prompt engineering | The instructions in the context | Ambiguity about the task, output shape, tone within limits | Missing knowledge; deeply ingrained behaviour | | RAG | The information in the context | Missing, private, changing, or citation-requiring knowledge | How the model writes; consistent structure | | Fine-tuning | The weights | Format, tone, domain idiom, reliable task-shape adherence | Facts that change; anything needing provenance |
That table is most of the answer. The rest of this post is why, and how to tell which row you are in.
The decision rule
Ask one question about your failures: is the model producing the wrong content, or the right content in the wrong shape?
If the model does not know something — a fact about your company, a product released after its training cutoff, the contents of a document — that is a knowledge problem. Retrieve.
If the model knows the answer but returns it as prose when you needed JSON, in the wrong register, at the wrong length, or without following your domain's conventions — that is a behaviour problem. Prompt first; fine-tune if prompting plateaus.
If the model is doing something reasonable but not what you meant — that is an instruction problem. Prompt.
Almost every real project is dominated by the first case, and a surprising number of teams respond to it by fine-tuning, which does not work well and takes ten times longer to find out.
Why fine-tuning is a bad way to add knowledge
This is the single most expensive misconception in applied LLM work, so it is worth being precise about the mechanism.
Fine-tuning adjusts weights so the model's output distribution shifts toward your examples. If you fine-tune on a thousand documents about your product, the model becomes more likely to produce text that sounds like your product documentation. It picks up vocabulary, phrasing, structure, and register very effectively.
Whether it reliably reproduces specific facts from those documents is a different matter, and the answer is: not well enough to depend on. Facts absorbed this way are diffuse. The model cannot tell you where one came from, cannot distinguish something it learned from you versus from pretraining, and will interpolate confidently between things it half-absorbed. You have made hallucinations more fluent and more plausible without making them less frequent.
Then there are the operational problems, which are worse:
Updating means retraining. A fact changes, and your options are another fine-tuning run or a stale model. With retrieval you change a document.
No provenance. You cannot cite a weight. Any application needing "according to this source" — legal, medical, financial, compliance, anything auditable — is ruled out immediately.
No access control. A fine-tuned model knows what it knows, for everyone. If some documents should only be visible to some users, this is a non-starter. Retrieval can filter by permission at query time.
Catastrophic forgetting. Aggressive fine-tuning on a narrow corpus degrades general capability. The model gets better at your domain and worse at reasoning, which is often a bad trade.
The one place knowledge-fine-tuning legitimately helps is a large, stable, closed domain where you need the model to have absorbed a specialised vocabulary — and even there it is usually a complement to retrieval, not a replacement.
What RAG is actually good at
Retrieval-augmented generation puts relevant documents into the context at query time and asks the model to answer from them.
It is the right choice when:
- The knowledge changes — prices, policies, inventory, current events. - The knowledge is private — internal documents, customer records, anything not in pretraining. - You need citations — the user must be able to check the source. - Access control matters — different users should see different subsets. - The corpus is large relative to what fits in a context window. - You need to add or remove a fact quickly, without a training run.
What it does not fix is how the model writes. A RAG system with excellent retrieval and a poor prompt produces well-grounded answers in the wrong format. Retrieval is about what is in the context; it has no opinion about the shape of the output.
RAG also introduces its own failure surface, which people underestimate. You now have a chunking strategy, an embedding model, a retrieval algorithm, possibly a reranker, and a prompt — and a failure anywhere in that chain looks identical from the outside. This is why RAG systems need evaluation harnesses that measure retrieval and generation separately; without that separation you cannot tell which component to fix and will usually guess wrong.
What fine-tuning is actually good at
Fine-tuning is genuinely excellent at things prompting struggles to make reliable:
Rigid output format. If you need valid structured output on every single call and prompting gets you to 97%, a fine-tune can close the gap more cheaply than retry logic and a parser.
Consistent tone and register. Matching a brand voice or a clinical documentation style is very hard to specify in words and easy to demonstrate with examples.
Domain idiom. How a radiology report is phrased, how a legal clause is structured, how a support ticket is triaged in your organisation. These are conventions with a thousand small rules nobody has written down.
Task shape adherence. A specific multi-step reasoning pattern you want followed every time, without restating it in every prompt.
Latency and cost reduction. A fine-tuned small model can often match a much larger prompted model on a narrow task. If you are running high volume, this is frequently the strongest argument for fine-tuning, and it has nothing to do with knowledge.
That last point deserves emphasis for anyone thinking about production economics. Fine-tuning as a compression technique — distilling a large model's behaviour on your specific task into a small one — is well-founded and underused. Fine-tuning as a knowledge technique is the misuse.
Start with prompting, and be honest about it
Prompting is the technique people skip past because it feels insufficiently serious.
It is free, instantaneous, and reversible. Before concluding it is not enough, make sure you have actually tried:
- Explicit output format with a concrete example of the exact shape you want. - Few-shot examples — three to five demonstrations, which is often the difference between unreliable and reliable. - Task decomposition — splitting one hard prompt into two easy ones, which frequently beats any amount of prompt polish. - Structured output constraints where the provider supports them, which solves format problems more reliably than either prompting or fine-tuning. - A clear role and scope statement, including what the model should refuse.
If you have not done all five, you do not yet know that prompting is insufficient. A large fraction of "we need to fine-tune" conclusions come from having tried a single unstructured prompt.
The signal that prompting has genuinely plateaued: your prompt is very long, adding to it now breaks something else, and performance on your evaluation set has stopped responding to changes. That is a real plateau. Reaching it takes a week, not an hour.
The cost comparison nobody puts in the table
Cost is usually presented as compute. That is the smallest term.
| | Prompt engineering | RAG | Fine-tuning | |---|---|---|---| | Time to first result | Hours | Days | Weeks | | Data required | A few examples | Your documents, chunked | Hundreds to thousands of labelled examples | | Ongoing maintenance | Edit text | Update documents, re-index | Retrain per change | | Cost to update a fact | Zero | Minutes | A training run | | Provenance | N/A | Native | None | | Per-request cost | Baseline | Higher (longer context) | Can be lower (smaller model) | | Latency | Baseline | Higher (retrieval step) | Can be lower | | Failure surface | Small | Large and multi-stage | Small at inference, large at training | | Hardest part | Knowing when to stop | Retrieval quality | Building the dataset |
The dominant cost of fine-tuning is not GPU time. It is producing several hundred high-quality labelled examples of the behaviour you want, which is slow human work and the reason most fine-tuning projects stall. Anyone comparing options on compute price alone is comparing the wrong column. If you do end up needing training hardware, choosing it deliberately matters more than buying the biggest available card.
Combining them is the normal production answer
The framing as a competition is the last thing to discard. Mature systems use two or three together, because they address different layers:
- Retrieval supplies the facts. - A fine-tune enforces the output shape and domain register. - The prompt carries the task instructions and constraints.
A concrete example: a clinical assistant needs current guidance (retrieval, with citations, because a clinician must be able to check the source), a consistent advisory register that never reads as a diagnosis (behaviour — prompting, and a fine-tune if prompting proves inconsistent), and explicit scope and refusal rules (instructions — prompt).
Trying to solve all three with any one technique produces a bad system. The clinical RAG assistant I built is retrieval-first for exactly this reason: the knowledge changes, it must be citable, and a fine-tuned model that could not point at its source would have been unusable regardless of quality.
How to diagnose which one you need
Do not decide from first principles. Measure.
Step one: build a small evaluation set. Thirty to a hundred realistic inputs with the output you wanted. This is unavoidable — without it every subsequent decision is a guess.
Step two: run your current setup and categorise failures. For each one, ask: did the model lack information, or did it have the information and use it wrongly?
Step three: read the distribution.
- Mostly missing information → retrieval problem. Build RAG. - Mostly wrong shape or tone → behaviour problem. Prompt harder, then fine-tune. - Mostly misunderstood the task → instruction problem. Fix the prompt. - Mixed → fix in that order: prompt, then retrieval, then fine-tune.
Step four: for suspected knowledge failures, run the oracle test. Paste the correct source document directly into the prompt and re-run. If the model now answers correctly, it is definitively a retrieval problem and fine-tuning will not help. If it still fails with the answer sitting in front of it, you have a reasoning or instruction problem, and better retrieval will not help either.
That test takes ten minutes and settles the argument more often than any amount of discussion. It is the single most useful diagnostic in this whole area.
Common mistakes
Fine-tuning to teach facts. Covered above; it is the big one.
Building RAG for a small static corpus. If everything fits in the context window and never changes, put it in the prompt. You do not need a vector database for forty pages.
Fine-tuning before having an evaluation set. You cannot tell whether it helped. Teams do this and then cannot answer whether to keep the fine-tune.
Treating RAG as a solved component. Chunking, embedding choice, hybrid retrieval, reranking — each is a real engineering decision. "We'll add RAG" is a project, not a step.
Choosing based on what sounds most impressive. Fine-tuning sounds like machine learning; retrieval sounds like plumbing. The plumbing is usually correct.
Closing
The question to ask is not:
"Should we use RAG or fine-tuning?"
It is:
"When it fails, is it missing information or misusing information it already has?"
The first is retrieval. The second is behaviour. You can answer it in an afternoon with thirty examples and the oracle test, and the answer will save you weeks.
The reason this matters more than it sounds: choosing wrong is not slightly suboptimal, it is a dead end. A fine-tune aimed at a knowledge problem produces a model that is more confident and no more correct, and you discover this after building the dataset.
If you are a student working this out for a project, or a team about to commit to one of these paths, get in touch — it is a question I am always happy to argue about.
Frequently asked questions
What is the difference between RAG and fine-tuning?
RAG changes what information is in the model's context at query time; fine-tuning changes the model's weights. In practice that means RAG is how you give a model knowledge it does not have — private, changing, or citable — and fine-tuning is how you change how the model behaves: its output format, tone, domain idiom, and adherence to a task shape. They fix different failures and are frequently used together.
Can fine-tuning teach an LLM new facts?
Not reliably enough to depend on. Fine-tuning shifts the output distribution toward your examples, so the model becomes good at sounding like your corpus, but specific facts are absorbed diffusely. The model cannot cite where a fact came from, cannot distinguish what it learned from you versus pretraining, and interpolates confidently between things it half-absorbed. It also cannot support access control, and updating a single fact requires another training run.
When should you use RAG instead of fine-tuning?
Use RAG when the knowledge changes, is private to your organisation, needs citations, requires per-user access control, or is too large for the context window. That covers the majority of real applications. Use fine-tuning when the problem is behavioural — you need a rigid output format, a consistent tone, a domain's writing conventions, or a smaller and cheaper model matching a larger one's behaviour on a narrow task.
Is prompt engineering enough on its own?
More often than people assume, and it should always be the first attempt because it is free and reversible. Before concluding otherwise, try an explicit output format with a concrete example, three to five few-shot demonstrations, decomposing one hard prompt into two easy ones, structured output constraints if your provider supports them, and a clear scope and refusal statement. A genuine plateau looks like a long prompt where additions break other behaviour and evaluation scores have stopped responding.
How much data do you need to fine-tune an LLM?
For behavioural fine-tuning, typically hundreds to a few thousand high-quality examples — and quality dominates quantity, since a few hundred consistent examples usually beat several thousand noisy ones. The real cost is not compute but the human work of producing that labelled set, which is where most fine-tuning projects stall. If you cannot produce several hundred examples of the behaviour you want, you are not ready to fine-tune.
Can you use RAG and fine-tuning together?
Yes, and it is the normal configuration in mature systems. Retrieval supplies the facts, the fine-tune enforces output shape and register, and the prompt carries task instructions and constraints. They operate on different layers, so combining them is composition rather than redundancy. Trying to solve knowledge, behaviour and instruction problems with any single technique produces a system that is mediocre at all three.
How do you know if your problem is retrieval or reasoning?
Run the oracle test. Paste the correct source document directly into the prompt and re-run the failing case. If the model now answers correctly, the problem is definitively retrieval and fine-tuning will not help. If it still fails with the answer sitting in front of it, the problem is reasoning or instructions, and improving retrieval will not help either. The test takes ten minutes and settles the question more reliably than discussion.
Related reading
- How to Evaluate a RAG System — And the Bug My Evaluation Caught — how to measure whether your retrieval is actually the problem. - From 50% to 100% Emergency Recall: Debugging Safety Routing in a Clinical RAG Assistant — a production RAG architecture in full, and why it is retrieval-first. - Two-Stage Chest X-Ray Triage: ResNet50 + YOLOv8 at 96.9% Accuracy — what fine-tuning looks like when the task genuinely calls for it.