Fine-Tuning Does Not Teach Facts, And That Changes The Decision

"We want the model to know our internal documentation, so we're going to fine-tune it on our docs."

That sentence describes a retrieval problem and proposes a behaviour solution, and it is the single most expensive misunderstanding in applied AI work.

Two different axes

Sort it by what you are actually dissatisfied with.

Retrieval-augmented generation adds knowledge. You fetch relevant material at request time and put it in front of the model. The knowledge lives in your store, not in the weights, so it can be updated the moment a document changes, permissioned per user, and — crucially — cited.

Fine-tuning adjusts behaviour. Format, tone, structure, adherence to a house style, reliable production of a particular schema, better performance on a narrow repetitive task. You are teaching the model how to respond, not what is true.

Knowledge, or form. Almost every real decision resolves once you name which one is bothering you.

Why fine-tuning on facts disappoints

Training on a corpus of documents does not install a lookup table. It shifts the distribution of what the model tends to produce — which is not the same as making specific facts reliably retrievable.

Worse, it can degrade the thing you were trying to fix. A model fine-tuned on your documentation becomes more fluent in the register and vocabulary of that documentation, which makes its confident inventions sound more authoritative, not less. You have improved the style of the hallucination.

And you get none of what makes retrieval operationally viable. There is no citation, so a user cannot verify an answer. There is no update path short of retraining. There is no per-user permissioning, so a model trained on documents some employees should not see now carries that content for everyone.

Where fine-tuning genuinely wins

It is not a bad technique. It is a technique for a different problem.

Reach for it when you need consistent structured output that prompting keeps almost getting right; when you need a specific voice or format across thousands of calls and the instructions to achieve it have grown unmanageably long; when a narrow, high-volume, repetitive task could run on a smaller and cheaper model if that model were specialised; or when latency and cost matter enough that trimming a long instruction preamble from every request pays for the training.

That last one is underrated. Fine-tuning can be a cost optimisation — moving behaviour from tokens you pay for on every call into weights you pay for once.

Try the cheap things first, in order

Prompting. A surprising share of "we need to fine-tune" turns out to be an instruction problem. Few-shot examples in particular do much of what people expect fine-tuning to do.

Retrieval. If the complaint contains the words "doesn't know about," this is your answer.

Both together. Retrieval for knowledge, fine-tuning for the shape of the response, is a coherent architecture and often the mature end state.

Fine-tuning alone. Rarely the right first move, and it demands a dataset you probably do not have yet — hundreds to thousands of properly formatted examples, curated, which is the real cost rather than the compute.

The question that decides it

Ask what a bad output looks like.

"It said something untrue about our product" — knowledge problem. Retrieval.

"It said the right thing in the wrong format, again" — behaviour problem. Prompting first, fine-tuning if that plateaus.

"It was right but took four paragraphs to say it" — behaviour.

"It confidently described a feature we removed last year" — knowledge, and specifically a freshness problem, which fine-tuning is structurally worst at because your weights are frozen at training time and your documents are not.

The dataset is the real cost

Teams budget for training compute and are surprised by what actually consumes the time.

Fine-tuning needs examples in a consistent format, enough of them, and — the part that hurts — curated. Scraped-together examples teach whatever inconsistency they contain, including the mistakes. If three people wrote your training examples to three different standards, you have taught the model to pick one at random.

That work does not disappear if the project fails, which is worth knowing before you start: a clean, well-formatted dataset of your own task is durable, and it doubles as evaluation material. The compute is the cheap, repeatable part.

Retrieval has an equivalent hidden cost, and it is chunking. How documents are split, what metadata travels with each chunk, and how results are ranked determines far more about output quality than the model does. Most disappointing retrieval systems are disappointing at the chunking layer, and swapping in a better model does not fix them.

Before you build either one

If you have not read the evaluation piece on this site, build the evaluation set first. Both routes are indistinguishable from expensive superstition without a way to measure whether the output improved.

And measure the boring baseline too. Plain prompting with good examples, over your existing retrieval, is the number both fancier options have to beat — and it wins more often than anyone planning a fine-tuning run expects.