Fine-tuning, RAG, or prompting?
Pick the wrong adaptation method and you can burn weeks fine-tuning a model to memorize facts it will still get wrong — when a day of RAG would have nailed it. There are three ways to bend a general base model to your task — prompting, RAG, and fine-tuning — and the whole decision comes down to knowing which one each problem needs, because they change completely different things.
Three methods, three different things they change.
Every way of adapting a model touches a different layer of the system. Get this mental model straight and the rest of the choice becomes almost mechanical:
- Prompting changes nothing in the model. You shape behavior purely through the instruction and the context you supply — including few-shot examples that demonstrate the pattern you want. The weights are untouched; you are just choosing what the model reads before it answers. See prompting basics.
- RAG changes the context, not the model. Retrieval-Augmented Generation leaves the weights exactly as they are and instead retrieves relevant external documents at query time and injects them into the context window. The model reads fresh material it was never trained on and answers from it.
- Fine-tuning changes the weights. You continue training the model on example input/output pairs, and the model's parameters actually move. This is training, not just configuration — you are producing a new version of the model that behaves differently by default.
The one-line version: prompting edits the instruction, RAG edits the context with fetched documents, fine-tuning edits the weights. Instruction, context, weights — three layers, three methods. Almost every "should I fine-tune this?" question dissolves once you ask which of those three layers the problem actually lives in.
What each one costs you.
The three methods sit on a ladder of effort, and the trade-off is consistent: the more you change, the more it costs up front and the more it can buy you at serving time.
- Prompting — instant, cheap, reversible. No training, no pipeline, no waiting; you edit text and try again. The limits are the context window (everything you want the model to know rides in the prompt on every call, costing tokens and latency each turn) and consistency — a prompt asks the model to behave a certain way, and it mostly will, but not with perfect reliability.
- RAG — moderate, ongoing effort. You build and maintain a retrieval pipeline plus a vector store (see chunking and vector search). The payoff is huge: knowledge updates without retraining — change the documents, not the model. The costs are added retrieval latency on every query and a context that can bloat if you inject too much.
- Fine-tuning — highest upfront cost, cheapest at serving time. You pay for data curation, a training run, and evaluation before you get anything back. But once the behavior is baked into the weights, it no longer needs to be re-specified in every prompt — so a fine-tuned model can run on shorter prompts, with lower inference latency and more consistent formatting than an equivalent prompted model doing the same job.
So the effort curve runs prompting < RAG < fine-tuning — but so does the leverage. Fine-tuning is the only one that can make the running system cheaper per call, which is exactly why it is worth the pain when a behavior is stable and high-volume.
Knowledge vs behavior — the distinction everything hinges on.
Here is the split that decides between RAG and fine-tuning, and it is the single most misunderstood point in this whole area. Ask what the problem actually is:
- Is it a knowledge gap? The model doesn't know something — your internal docs, a customer's order history, this week's prices, a fact that changes often. That is a job for RAG. You supply the knowledge at query time, and it stays current because you update the documents, not the model.
- Is it a behavior gap? The model knows plenty but doesn't act the way you need — the wrong tone, the wrong output format, a house style, a specialized task it keeps fumbling. That is a job for fine-tuning. You bake the behavior in so it becomes the default.
Fine-tuning is not how you teach a model new facts. This is the correction that saves the most wasted effort. Fine-tuning primarily teaches form, style, and behavior — not reliable recall of new information. Trying to inject facts by fine-tuning is data-hungry, goes stale the moment the facts change, and — as a documented research finding, not folklore — can actually increase hallucination, because you are teaching the model the confident shape of an answer without reliably teaching the answer. When you need the model to know something, reach for RAG, which stays updatable without retraining.
Say the tell out loud: "teach it what to know" points to RAG; "teach it how to act" points to fine-tuning. Most real projects have some of both — and, crucially, that does not mean you must choose.
The order to try them, and why it is not either/or.
The field's default sequence is deliberately cheapest-first:
- Start with prompting. It is free and instant; a surprising amount of "we need to fine-tune" turns out to be a prompt that was never written carefully.
- Add RAG when you hit a knowledge or freshness wall. The moment the gap is facts the model doesn't have, retrieval is the answer.
- Fine-tune last — for behavior, format, style, or serving-time latency — and only after prompting and RAG have plateaued. It is the biggest investment, so you spend it only once cheaper moves stop paying off.
RAG and fine-tuning are complementary, not competitors. They solve different problems — knowledge vs behavior — so "pick one" is a false choice. Production systems very often run a fine-tuned model that also retrieves: the fine-tune fixes how it answers, RAG keeps what it answers about current. Reaching for fine-tuning does not mean giving up retrieval.
One reason fine-tuning is now reachable at all: you rarely retrain the whole model. Full fine-tuning updates every weight and is expensive, so most people use PEFT (Parameter-Efficient Fine-Tuning), which freezes the base model and trains a small number of added parameters. LoRA (Low-Rank Adaptation) is the dominant PEFT method, and QLoRA combines LoRA with quantization to fine-tune on modest hardware — together they turned fine-tuning from a data-center project into something you can do without ever touching the billions of frozen weights underneath.
When you are ready to go deeper, the wiki's training-agentic-models deep-dives take fine-tuning into production — prompt, fine-tune, or RL, SFT, rejection sampling, and distillation, and RLHF and RLAIF — while the retrieval-and-rag deep-dives do the same for retrieval, in advanced RAG architectures and evaluating RAG.