RAG vs fine-tuning: a decision framework.

Almost every "should we fine-tune?" conversation is really a knowledge problem wearing a behaviour problem's clothes — or the reverse. RAG (retrieval-augmented generation) and fine-tuning are not competing answers to one question; they answer two different questions. Confuse them and you spend a quarter teaching a model facts it will forget, or curating a corpus to fix a tone it could never learn that way. This is the framework we use before writing a line of code.

Start with mechanics, because the trade-offs fall straight out of them. RAG leaves the model's weights untouched and instead changes what the model *sees* at inference time: you retrieve relevant documents, paste them into the context window, and let the model reason over them. Fine-tuning changes the model's *weights* — you continue training on curated examples so the model internalises a behaviour or a style.

• RAG = supply knowledge at runtime. The model reads your data fresh on every request. Update the data, and the next answer reflects it instantly — no retraining.
• Fine-tuning = shift behaviour at training time. You bake a pattern (format, tone, classification rule, reasoning style) into the weights. The behaviour is "free" at inference; the model no longer needs lengthy instructions to produce it.
• Prompt engineering is the cheap third option people forget. A well-structured system prompt — examples, role, constraints — often closes the gap that someone assumed required fine-tuning.

The crisp test: if the answer changes when your data changes, you have a retrieval problem. If the answer changes when your *standards* change — how you want it written, classified, or structured — you have a behaviour problem. The first points at RAG; the second points at fine-tuning, but only after prompting has hit its ceiling.

Reach for RAG when the model needs facts it was never trained on, or facts that change faster than any training cycle could keep up with. Retrieval also gives you the property regulated teams care about most: every answer can cite the source passage it came from.

• Knowledge is proprietary or private — internal docs, tickets, contracts, a product catalogue the base model has never seen.
• Knowledge changes often — pricing, policy, inventory, the latest release notes. Re-index, do not retrain.
• You need attribution — answers must link back to a clause or a document for trust, audit, or compliance.
• The corpus is large and sparse — you only need a handful of relevant passages per query, not the whole library memorised.

RAG does not teach the model *how* to behave. If retrieval surfaces the right passage but the model still answers in the wrong format or misclassifies it, that is a behaviour gap — and no amount of better retrieval fixes it. Note too that a clean tool boundary, often an MCP server, is what makes retrieval auditable instead of a black box.

Fine-tuning is for behaviour you can demonstrate but struggle to describe — and that you need consistently, at volume, without burning context on instructions every call. It shines on narrow, repetitive, well-defined tasks where you have (or can build) a high-quality labelled set.

• A consistent output format or house style that long prompts only approximate — and you have hundreds-plus clean examples of it done right.
• A specialised classification or extraction task where the rules are tacit and easier shown than stated.
• Latency and cost pressure at scale — a fine-tuned model needs shorter prompts, which trims tokens on every one of millions of calls.
• A reasoning or domain style the base model can be nudged toward, where prompting plateaus before the quality bar.

What fine-tuning does *not* do: inject fresh facts reliably. Train a model on last quarter's policy and it answers from last quarter's policy with total confidence — there is no source to cite and no easy way to correct one wrong fact short of curating data and retraining. If the knowledge moves, fine-tuning fights you.

The honest answer for most production systems is "both, but staged." Get retrieval and prompting right first; it is faster, cheaper, and tells you whether you even have a behaviour problem left to fine-tune away. Map your situation against the table before committing to a pipeline.

When you do run both, the division of labour is clean: RAG owns *what* the model knows, fine-tuning owns *how* it responds. Build them as separable layers so you can re-index without retraining and retrain without re-indexing.

Most teams compare build cost and stop there. The deciding cost is maintenance — the bill that recurs every month after launch. We talk in tiers rather than figures, because the absolute numbers drift with every model release.

• RAG build: moderate — retrieval, chunking, an index, an evaluation harness. RAG maintenance: ongoing-but-cheap — keep the index fresh and the chunking honest; no retraining.
• Fine-tune build: lower up front if your data is clean — but data curation is the real work and it is rarely cheap. Fine-tune maintenance: lumpy — every meaningful behaviour change is another curation-plus-retrain cycle.
• Per-call cost: RAG spends more tokens (you pay to read context each time); a fine-tuned model spends fewer (shorter prompts). At high volume that flips the total-cost picture — which is exactly why latency-bound, high-traffic tasks lean fine-tune.
• Hidden cost of fine-tuning: model lock-in. A new base model means re-running your fine-tune; a RAG pipeline usually just points at the new model and keeps working.

Neither technique is decidable by intuition. The only way to know whether prompting plateaued, retrieval missed, or behaviour drifted is to measure — which means a labelled eval set with the failure modes each technique produces, separated so you can read them.

1. Build the eval set first — representative inputs with known-good outputs, including the edge cases that actually hurt in production.
2. Score retrieval and generation separately. A wrong answer from the right passage is a behaviour bug; a wrong answer from a missing passage is a retrieval bug. Conflate them and you will tune the wrong layer.
3. Establish a prompt-only baseline before any pipeline or training. It is the cheapest intervention and frequently the surprise winner.
4. Re-run the same eval after every change — new index, new fine-tune, new base model — so quality is a number you watch, not a vibe you hope for.

for case in eval_set:    retrieved = retrieve(case.query)            # RAG layer    answer = generate(case.query, retrieved)    # behaviour layer     record(case.id, {        # did retrieval surface the passage we needed?        "retrieval_hit": case.gold_doc_id in ids(retrieved),        # given context, did the model answer correctly + in-format?        "answer_correct": judge(answer, case.gold_answer),    }) # low retrieval_hit  -> fix the index / chunking (a RAG problem)# high hit, low correct -> fix behaviour (prompt, then fine-tune)

Read top-to-bottom, those two columns tell you which lever to pull. Eval-driven development is the whole discipline of our AI workflow automation work — we do not ship an AI feature without the harness that proves it, and keeps proving it after launch.

Run the questions in order. Stop at the first one whose answer points somewhere — most systems never reach the bottom.

This is a discovery-call-length decision, not a quarter-long one — once the eval set exists. We build Claude-native by default and run a prompt-first, eval-driven loop: baseline, then retrieval, then fine-tune only if the numbers still demand it. If you want the framework applied to your system instead of read about, that is exactly what an AI workflow automation engagement is for, and our pricing is built around two-week cycles you can stop after.