The returns assistant passed every test. It was fine-tuned on all twelve of Brindlemoor Goods' policy documents, and it answers in the house three-line format. Then the return window moved from 30 days to 45, and it kept saying 30.
(Brindlemoor is a fictional shop, and every number in this post is illustrative.)
Nothing in the model told you it was wrong, because nothing in the model knew where the 30 came from. It had no policy page to point at and no date on the fact. It just said 30, in a clean three-line reply that ended with the policy name, exactly as trained.
This post is about why that happens, and a rule for deciding which parts of your assistant belong in the weights and which belong in the prompt. The short version: fine-tuning changes how a model behaves far more reliably than what it knows.
What a fine-tuning run actually changes
Fine-tuning means continuing to train a pretrained model on your own examples. The weights are the learned numbers inside the model, and a run nudges them so the model's outputs move closer to your examples.
What moves depends on what your examples have in common. If all 400 training answers are three lines and end with a policy name, that pattern is the strongest signal in the data. The model picks it up quickly and holds it across topics it never saw in training. Format, tone, and the shape of the task are what fine-tuning tends to change most reliably.
Facts are different. Suppose 6 of those 400 answers mention the 30-day window. The model sees that number a handful of times, in a handful of phrasings, next to a lot of other text. In practice it usually absorbs the fact loosely: it may get it right when the question resembles a training example, and drift when the wording changes. Loosely held is the important part. A fact the model only half-holds still comes out sounding fully sure.
This is the trap. Training on your domain makes the model sound like it belongs there, and sounding right is not evidence of being right. A base model that says "I'm not sure about Brindlemoor's policy" is at least honest about it. The fine-tuned one has learned the house style of certainty.
None of this is a law. Results vary with the model, the amount of data, and how the facts are phrased. As a rule of thumb, fine-tuning teaches behaviour reliably and facts only patchily.
Why facts are the wrong thing to put in weights
Even if a run absorbed every fact perfectly, you would still pay two costs. And when it does not absorb them, there is a third.
Facts go stale the day they change. The window moves to 45 days in Q3. To fix a fine-tuned assistant you need new training examples that state the new window, a new run, and a fresh check that the old number is really gone and nothing else moved. With the fact in a prompt, you edit one line in one document. That gap is a cost you pay every time a policy, price, or rota changes.
There is no source to point at. A support lead asks, "Where did it get 30 days?" With retrieval, the answer is a document and a paragraph. With weights, the honest answer is "from training, somewhere". For anything a customer might dispute, or an auditor might ask about, an answer with no citation is a liability, however correct it happens to be.
When the facts are not absorbed, training can push the model to guess. Say your dataset has a question about the extended-warranty return rules, and the model never really absorbed those rules. Training still rewards it for producing a confident, well-formed answer. Do that across enough examples and the model can learn that answering is what it does, whether or not it holds the fact. This is one way fine-tuning on new facts can increase hallucination: a fluent answer that nothing true supports. It is a risk to watch for, not a certainty, and you can measure it on your own data before you trust it.
Retrieval (often called RAG, retrieval-augmented generation) avoids all three. At question time you fetch the relevant policy text and put it in the prompt, so the model reads the fact instead of remembering it. For facts that change, that is usually the better default.
The Change-Rate Rule: put each requirement where changing it is cheapest
Here is a way to decide in a meeting. For each requirement on the assistant, ask how often it changes, and put it where changing it is cheapest.
Walk Brindlemoor's requirements through the rule:
Requirement type | Brindlemoor example | Lever | Why |
|---|---|---|---|
Stable format | Three-line reply ending with the policy name | Fine-tune | Never changes, and the model should follow it without being reminded |
Changing fact | Return window: 30 days, then 45 in Q3 | Retrieve | Changes on a schedule and needs a citable source |
Stable jargon | The support team's shorthand for product families | Fine-tune, if volume justifies it | Stable, and used in almost every question |
Per-user data | This customer's order date and item | Prompt | Different on every request, so it can only arrive with the request |
The two rows at the top of the table decide most cases. To see the difference in the training data itself, here are two illustrative rows (not a training recipe, and shown as pretty-printed for readability; real JSONL has one object per line):
{"messages": [
{"role": "user", "content": "Policy excerpt: Standard Returns. Window: 30 days from delivery. Item must be unused.\n\nQuestion: Can I return a lamp I got 10 days ago?"},
{"role": "assistant", "content": "Yes, the lamp is inside the return window.\nKeep it unused and in its box.\nPolicy: Standard Returns"}
]}
{"messages": [
{"role": "user", "content": "How long do I have to return something?"},
{"role": "assistant", "content": "30 days from delivery.\nThe item must be unused.\nPolicy: Standard Returns"}
]}
The first row supplies the policy in the prompt and teaches the model what to do with it: read the excerpt, answer in three lines, name the policy. When the window becomes 45 days, this row stays valid. The second row has no excerpt, so the only place the 30 can come from is the weights. When the window changes, this row is wrong, and so is every model trained on it.
If you fine-tune, make your dataset look like the first row.
When fine-tuning on facts does make sense
The rule has exceptions, and they are worth naming so you can tell when you are in one.
Stable, high-volume domain vocabulary. If your assistant works in a field with its own dense language (part numbers, clinical shorthand, an internal product taxonomy) and that language rarely changes, training on it can help the model read and write it fluently. Notice that this is closer to teaching a dialect than teaching facts. The vocabulary is stable, so the retraining gap rarely comes up.
Hard latency or context limits. Retrieval adds a step and puts extra text in every request. If you have a strict response-time budget, or a context window (the amount of text a model can read in one request) too small for what you would need to send, moving some knowledge into the weights can be a fair trade. Be careful with this one, because it ages fast. Twelve policy documents are a modest amount of text and many models accept far more than that in one request, so Brindlemoor is not in this situation. Check the current limits for your model before you argue from them.
What would have to be true for fine-tuning on facts to be defensible:
The facts stay put longer than your retraining cycle, ideally for good.
A wrong answer is cheap, or something else (a human, a rule, a lookup) checks the answer before it reaches anyone.
You can test recall on questions phrased differently from your training data, so you know what the model holds and what it is faking.
You have a repeatable way to retrain when something does change.
If you cannot tick all four, the facts belong somewhere else.
The hybrid: fine-tune for behaviour, retrieve for facts
For most teams the practical answer is both, each doing what it is good at. It comes down to which requirement lives where. No exotic architecture is needed.
For Brindlemoor, that split looks like this:
In the weights: the three-line format, the polite but brief tone, the habit of ending with the policy name, and the instruction to say "I don't have that policy" when no excerpt is supplied.
In retrieved context: the current text of the relevant policy, whether the window is 30 days or 45.
In the prompt for this request: the customer's order date, the item, and today's date.
Now the Q3 change is an edit to one policy document. The fine-tuned behaviour does not need to know. It reads whatever excerpt arrives and formats the answer the way it always has.
Notice the last weights item. Teaching the model to decline when no excerpt is present is behaviour, not fact, and it directly counters the confident-guessing problem. If your training data includes examples where the right answer is "I don't have that", the model has at least seen that refusing is allowed.
Takeaway
Before you fine-tune on your company's documents, run each requirement through three questions.
Does it change? If it changes on a schedule, like a return window, price list, or on-call rota, it goes in retrieved context, not weights. The fix should be a one-line edit, not a training run.
Does the answer need a source? If someone might ask "where did that come from?", retrieval can show them and weights cannot.
Is it behaviour or knowledge? Format, tone, task shape, and stable jargon are what fine-tuning changes most reliably. Put those in the weights and build the dataset so each row teaches a behaviour, not a fact. Anything that differs on every request, like an order date or today's date, goes in the prompt.
Before you trust any fine-tuned assistant, ask it the return-window question three different ways once the window is 45 days. If it says 30 to any of them, the fact was in the wrong place.
