Fine-Tuning

Training a language model from scratch is far beyond most organizations. Fine-tuning starts from a model somebody else already trained and nudges it with a comparatively small set of your own examples — a few hundred to a few thousand — so it learns your format, your house style, or the conventions of your field.

Fine-tuning changes the model's weights, which makes it good at teaching *behavior*: respond in this structure, adopt this tone, classify into these categories. It is much weaker at teaching *facts*. Knowledge baked in through fine-tuning is hard to update and hard to verify, which is why retrieval is usually the better tool for information that changes.

Fine-tuning is frequently proposed as the answer to "the model doesn't know our business", and it is usually the wrong answer. It is slow, it needs curated examples, it has to be redone when you change base models, and it makes errors harder to trace. Getting the choice right saves real money.

Rule of thumb: if the problem is the model doesn't *know* something, use retrieval. If the problem is the model doesn't *behave* the way you need, consider fine-tuning — but try a better prompt and a few examples in the context first, which costs nothing and often closes the gap.