Three tools get reached for when a model is not doing what you want: prompting, RAG, and fine-tuning. They are not interchangeable, and choosing wrong wastes time and money — most often by fine-tuning when a better prompt or a retrieval step would have solved it in an afternoon. This guide lays out what each one actually changes, what it costs, what it fails at, and a simple way to decide.

What each one changes

The cleanest way to hold these apart is by what they touch. Prompting changes the instructions you give a fixed model — you are steering behavior at request time, changing nothing permanent. RAG changes the knowledge available to the model — it injects relevant external information into the prompt so the model can answer from material it was never trained on. Fine-tuning changes the model itself — you continue training it on examples so a new behavior or style is baked into the weights. A slogan that holds up: prompt to steer, retrieve for knowledge, fine-tune for behavior.

Prompting: the first and cheapest move

Prompting is free, instant, and reversible. You change some words and immediately see the result; nothing is committed. It is the right first move for almost everything, and modern models are steerable enough that a clear, well-structured prompt with a good example handles a surprising range of tasks that people assume need training.

Its limits are real, though. Prompting cannot supply knowledge the model does not have, it cannot guarantee a rigid output format across thousands of calls with zero drift, and if you find yourself pasting the same long instructions into every request, you have outgrown ad-hoc prompting — though the fix for that is often a reusable system prompt, not training. Prompting fails when the gap is missing information or a hard consistency requirement, not phrasing.

RAG: when the gap is knowledge

Reach for RAG when the problem is that the model does not know something — your documents, your data, current facts, anything past its cutoff. Its costs are the engineering to build and maintain the pipeline (chunking, embedding, a vector store, retrieval quality) and a little extra latency and token cost per call to include the retrieved passages. In exchange you get answers grounded in your material, the ability to cite sources, and instant updates: change a document, re-embed it, done.

RAG fails when the problem is not knowledge. It will not make a model adopt a consistent tone, follow a rigid format, or acquire a skill — it adds facts to a prompt, nothing more. And it is only as good as its retrieval; surface the wrong passages and the model answers confidently from the wrong material. It is covered in depth in the RAG guide, but the one-line test is this: is the model missing information it could simply read? Then RAG.

Fine-tuning: when the gap is behavior

Fine-tuning continues training a base model on your examples so a behavior becomes intrinsic. It is the right tool for a specific, repeated behavior: a rigid output format that must hold at scale, a consistent voice or style, a specialized classification task, or teaching a smaller, cheaper model to imitate the outputs of a larger one so you can run it for less. Most fine-tuning today is not full retraining but lightweight adapters — LoRA, and its memory-frugal variant QLoRA — that train a small set of extra parameters on top of a frozen base model, which is why it can now be done affordably, sometimes on a single consumer GPU.

The costs are the highest of the three. You need a quality dataset of examples — often hundreds to thousands, and assembling good ones is the real work. You need a training run and a way to evaluate the result. And you take on maintenance: a fine-tune is frozen against a particular base model, so when a better base model ships, your adapter does not automatically benefit, and you may face redoing the work. Fine-tuning fails, expensively, when it is used to add knowledge — which brings us to the most common mistake.

The wrong reasons to fine-tune

  • To teach the model facts. The most frequent error. Fine-tuning on your documents does not reliably make a model recall them, and it happily makes up plausible-wrong details in the same style. Facts belong in retrieval, not the weights.
  • Because the prompt felt like too much effort. People jump to training to avoid iterating on a prompt, then spend far more time assembling a dataset than a good prompt would have taken. Exhaust prompting first.
  • To keep information current. A fine-tune is a snapshot; the moment your data changes it is stale. Anything that updates wants retrieval.
  • For a one-off or low-volume task. The dataset-plus-training overhead only pays back at scale, or where a rigid behavior genuinely cannot be prompted.
  • To fix occasional wrong answers. Sprinkling a few corrections into training rarely fixes a reasoning gap and can quietly degrade other behavior.

The underrated middle: system prompts and few-shot

Between a throwaway prompt and a training run sits the option most people skip. A system prompt — a fixed instruction block applied to every request — captures persistent rules, role, tone, and constraints without any training, and solves most I-keep-repeating-myself problems on its own. Pair it with few-shot examples, a handful of input-output pairs that demonstrate exactly the format and style you want, and you can pin down behavior that people assume requires fine-tuning. This middle tier is cheap, immediate, and fully reversible, and it should be exhausted before anyone opens a training script. A large fraction of we-need-to-fine-tune turns out to be we-needed-a-solid-system-prompt-and-three-good-examples.

A simple decision guide

  • Start by prompting. Rewrite the request, add structure, add an example. Most problems end here.
  • Repeating the same rules every time? Move them into a system prompt and add a few-shot example or two before considering anything heavier.
  • Is the model missing information it could read? Add RAG. This is a knowledge gap, not a behavior gap.
  • Do you need a rigid behavior, format, or voice — or a cheaper specialized model — at real volume, and prompting has genuinely plateaued? Now consider fine-tuning, ideally a LoRA adapter on a strong base model, kept alongside retrieval rather than replacing it.
  • Assume you will combine them. The systems that work best in practice are hybrids: a solid system prompt, retrieval for what changes, and — only where justified — a light fine-tune for behavior that must be baked in.

The sequence that saves the most pain is prompt, then retrieve, then fine-tune, in that order, stopping as soon as the problem is solved. Diagnose the gap honestly first — phrasing, knowledge, or behavior — and match the tool to it. Reaching for the heaviest tool first is the expensive way to learn what a system prompt could have done.