Reinforcement Learning from Human Feedback

After pre-training, a model is knowledgeable but not helpful. RLHF closes that gap. Humans compare pairs of model outputs and indicate which is better; those judgments train a reward model that predicts human preference; the language model is then optimized against that reward.

This is the step that produces the behavior people associate with AI assistants: answering the question asked, declining harmful requests, adopting a consistent tone. It is arguably the single most commercially important development in the field, because it made capable models usable by non-specialists.

It also introduces characteristic failure modes. Optimizing for what raters prefer can produce sycophancy — agreeing with the user, hedging, and padding answers, because raters reward those. Variants like reinforcement learning from AI feedback replace some human raters with models to cut cost.

RLHF is where a model's values and manner are set, so it is where a great deal of the alignment debate actually lands. It also means personality is a trained artifact rather than an emergent trait — and that vendors can and do change it between versions.

Recognize sycophancy as a trained tendency rather than agreement. If a model reverses position the moment you push back, that is RLHF, not new reasoning — ask it to argue the opposite case to get a real read.