"We want the model to know our internal documentation, so we're going to fine-tune it on our docs."

That plan may help some tasks, but it leaves an important question unanswered: is the failure caused by unavailable evidence, poor retrieval, weak use of context, or the behaviour learned by the model? Those causes call for different experiments. Training can change factual knowledge as well as style; it does not automatically provide a dependable, current, permission-aware document lookup.

Two different axes

Sort it by what you are actually dissatisfied with.

Retrieval-augmented generation supplies external evidence at request time. An application fetches material and gives it to the model. The original RAG paper combines parametric memory with a retrieved index; it also trains parts of that combined system. Retrieval and fine-tuning are therefore compatible techniques, not exclusive product categories.

Fine-tuning changes model parameters for a task or domain. It can improve formats, task performance, and factual associations, depending on the data and objective. The question is whether those changes solve the measured failure better than simpler interventions. Parameter updates alone do not supply a source citation or enforce a reader’s access rights.

Use evidence availability and task behaviour as diagnostic questions, rather than a strict knowledge-versus-form rule. A model may have the right passage and still answer incorrectly, or produce the requested format while relying on an outdated fact. Inspect those cases separately.

Why training is not a document lookup guarantee

Training on a corpus of documents does not install a lookup table. It shifts the distribution of what the model tends to produce — which is not the same as making specific facts reliably retrievable.

After training, fluent domain language can still accompany an unsupported answer. That is a failure to test for, not a claim that fine-tuning inevitably makes hallucination worse. Check answers against held-out questions and authoritative passages, including cases where the correct response is uncertainty or refusal.

Retrieval gives an application an explicit place to implement source updates, access checks, and citations. None is automatic: ingestion and caches can be stale, retrieval can select the wrong passage, and generated citations can be inaccurate. Enforce permissions before exposing chunks, check that cited material supports the claim, and test access revocation. A fine-tuned system can use these same application controls when combined with retrieval.

Where fine-tuning genuinely wins

Fine-tuning is a candidate when you have a defined task, appropriate training rights, consistent examples, and an evaluation that can distinguish improvement from memorisation. Its value depends on the task rather than a universal preference for one architecture.

Reach for it when you need consistent structured output that prompting keeps almost getting right; when you need a specific voice or format across thousands of calls and the instructions to achieve it have grown unmanageably long; when a narrow, high-volume, repetitive task could run on a smaller and cheaper model if that model were specialised; or when latency and cost matter enough that trimming a long instruction preamble from every request pays for the training.

Reducing repeated instruction tokens can lower serving cost, but training, data preparation, evaluation, model storage, and ongoing updates also have costs. Measure the complete workload and repeat the calculation when the model or request mix changes. There is no guaranteed saving from moving instructions into weights.

Try the cheap things first, in order

Prompting. Establish a clear instruction and few-shot baseline, with the same relevant context available to each candidate. Check whether the failure persists before adding infrastructure.

Retrieval. When required evidence is missing or stale, test whether the right documents can be found and supplied within access and latency constraints. If the right passage is already present, investigate context use and answer generation too.

Both together. A retrieved evidence store and a model trained to use that context can be evaluated as one system. Compare their contribution separately so a better retriever is not mistaken for a better fine-tune.

Fine-tuning alone. Test this when a stable, bounded task has a suitable training set and does not require live lookup for every answer. There is no universal minimum example count; record the dataset, quality checks, training objective, and why the resulting model meets the task.

The question that decides it

Ask what a bad output looks like.

"It said something untrue about our product" — check the source, retrieval result, context, and generated answer before choosing a remedy.

"It said the right thing in the wrong format, again" — behaviour problem. Prompting first, fine-tuning if that plateaus.

"It was right but took four paragraphs to say it" — behaviour.

"It confidently described a feature we removed last year" — inspect index freshness, cache expiry, evidence selection, and model reliance on old knowledge. Updating one layer may leave the others stale.

The dataset is the real cost

Teams budget for training compute and are surprised by what actually consumes the time.

Fine-tuning needs consistent, appropriately licensed examples and labels. Decide what behaviour each example teaches, remove avoidable contradictions, and preserve the dataset version. If annotators disagree, resolve the rubric or represent that uncertainty before interpreting the model’s score.

Keep training, development, and final evaluation partitions separate. A curated collection can supply each partition, but the same examples should not double as both training evidence and an independent test. Check near-duplicates, related documents, and repeated entities when those could leak the answer. Report which data influenced tuning and which remained held out.

Retrieval also has operational costs: ingestion, chunking, metadata, ranking, access filtering, context limits, and citation validation. Test each stage against real failures rather than assuming chunking is always the dominant cause. A stronger generator may help context use, but cannot recover a required fact that neither its knowledge nor its supplied evidence contains reliably.

Before you build either one

If you have not read the evaluation piece on this site, build the evaluation set first. Both routes are indistinguishable from expensive superstition without a way to measure whether the output improved.

Measure a plain prompting baseline and, where relevant, the existing retrieval path. Compare accuracy on held-out cases, supported citations, denied-access behaviour, latency, and cost. Publish the cases that still fail and the conditions under which you would reverse the decision. This guide supplies an evaluation method, not a benchmark claiming one route wins for your workload.