Fine-Tuning

Training a language model from scratch is far beyond most organizations. Fine-tuning starts from a model somebody else already trained and nudges it with a comparatively small set of your own examples — a few hundred to a few thousand — so it learns your format, your house style, or the conventions of your field.

Fine-tuning changes the model's weights, which makes it good at teaching behavior: respond in this structure, adopt this tone, classify into these categories. It is much weaker at teaching facts. Knowledge baked in through fine-tuning is hard to update and hard to verify, which is why retrieval is usually the better tool for information that changes.

Fine-tuning is frequently proposed as the answer to "the model doesn't know our business", and it is usually the wrong answer. It is slow, it needs curated examples, it has to be redone when you change base models, and it makes errors harder to trace. Getting the choice right saves real money.

That it is how you give a model your knowledge. This is the most common and most costly misconception, because it sends teams down a long path toward the wrong tool. Fine-tuning is good at teaching behavior — a format, a tone, a classification scheme. It is weak at teaching facts: knowledge baked into weights is hard to update, impossible to cite, and leaves you no way to tell whether an answer came from your data or from the model's training. If the problem is that the model does not know something, use retrieval.

That it is expensive and out of reach. It was; LoRA and related methods changed the economics by orders of magnitude. Rather than updating every parameter, they train a small add-on — typically a fraction of a percent of the model — with results close enough to full fine-tuning for most purposes. If you priced this out a couple of years ago and concluded it was infeasible, price it again.

That it is the first thing to try. It is usually the third. Better context and a few examples in the prompt cost nothing and often close the gap entirely; retrieval handles knowledge; fine-tuning is for when the model still will not behave the way you need after both.

Rule of thumb: if the problem is the model doesn't know something, use retrieval. If the problem is the model doesn't behave the way you need, consider fine-tuning — but try a better prompt and a few examples in the context first, which costs nothing and often closes the gap.