RLHF and Preference Optimization — Post-Training & Alignment

A language model starts as a machine for predicting the next token, not as a machine for being helpful, honest, safe, concise, or pleasant to work with. Reinforcement learning from human feedback, usually shortened to RLHF, is one way to close that gap: people express preferences about model behavior, and those preferences become training signals. Preference optimization is the broader family of methods that uses those signals to shape a model toward responses people are more likely to choose.

The basic problem: prediction is not preference

A large language model is first trained on a simple task: given some text, predict what text comes next. That task is powerful because it teaches grammar, facts, patterns of reasoning, styles of writing, and the shape of many human activities. But it does not, by itself, teach the model what answer a user would prefer in a specific moment.

Consider the prompt, Explain how to reset a password. Many continuations are statistically plausible. One might be too terse, one might be too long, one might invent a product-specific step, and one might be clear and cautious. Next-token prediction can make all of them fluent. It does not automatically know which one is best for this user, under these instructions, with these safety constraints.

RLHF reframes the problem. Instead of asking only, What text is likely, it asks, Which response would people prefer. That shift sounds small, but it changes the training signal from imitation to judgment.

What human feedback actually looks like

Human feedback rarely means a person writing a perfect answer from scratch. More often, it means a person comparing two or more model responses to the same prompt and choosing the better one. The prompt might ask for a summary, a plan, a coding answer, a customer support reply, or a refusal to an unsafe request.

The comparison can be simple: Response A is better than response B. It can also be richer: A is more accurate, B is safer, A follows the requested format, B is too verbose. The important point is that the feedback is relative. People are often more consistent when choosing between options than when assigning an absolute score from 1 to 10.

Good preference data needs clear criteria. If one reviewer rewards friendliness, another rewards brevity, and a third rewards aggressive refusal, the model receives a noisy lesson. A useful feedback program defines the target behavior before collecting labels: helpfulness, correctness, calibration, tone, safety, domain policy, and format fidelity.

A prompt gives the model a task or situation.
Several candidate responses are generated.
A reviewer ranks or chooses among the responses.
The choice is stored as preference data for training or evaluation.

The classic RLHF pipeline

The classic RLHF pipeline has three broad stages. First, a base model is trained or adapted to follow instructions. This is often called supervised fine-tuning, where the model learns from examples of prompts and good responses. Second, people compare model outputs, and those comparisons are used to train a reward model. Third, the language model is optimized to produce responses that the reward model scores highly.

A reward model is a smaller judge trained to predict human preference. Given a prompt and a response, it outputs a score. It is not the final assistant; it is a training tool that approximates the pattern in human choices. Because asking people to judge every possible model update would be too slow and expensive, the reward model stands in for human feedback during optimization.

The reinforcement learning part comes from the idea of actions and rewards. In a game, an agent takes actions and receives points. In RLHF for language models, the model produces tokens and receives a reward score for the completed response. Training adjusts the model so that future responses are more likely to receive high scores, while usually adding a constraint that prevents it from drifting too far from the original model.

RLHF runs in three stages: supervised fine-tuning, training a reward model, then optimizing against it.

Preference optimization without a separate reward model

RLHF is influential, but it is not the only way to learn from preferences. Preference optimization is the broader category. Some methods train the model directly on chosen and rejected responses without first building a separate reward model or running a full reinforcement learning loop.

Direct preference optimization, or DPO, is the best-known example. Instead of training a reward model and then using it to guide another training process, DPO updates the model so that chosen responses become more likely and rejected responses become less likely, relative to a reference model. The reference model matters because it keeps training anchored; without an anchor, the model can overfit to quirks in the preference data.

The practical appeal is simplicity. Direct methods can be easier to implement, easier to reproduce, and less fragile than reinforcement learning pipelines. The tradeoff is that they still depend on the quality, coverage, and consistency of the preference data. A simpler optimizer does not fix a confused definition of what better means.

📌

Names vary. RLHF is often used loosely, but many modern systems use preference optimization methods that are not classic reinforcement learning.

RLHF trains a reward model and runs a reinforcement loop; DPO learns directly from chosen and rejected pairs.

What can go wrong

The central risk is confusing the proxy with the goal. A reward model is a proxy for human preference, not human preference itself. If the language model learns how to exploit that proxy, it can produce responses that score well but are not actually better. This is called reward hacking.

Reward hacking can be subtle. The model may learn to sound confident, add unnecessary caveats, flatter the user, overuse a favored format, or refuse harmless requests because refusals were over-rewarded in safety data. These behaviors can look aligned on a narrow metric while making the product worse.

Another risk is preference collapse. If reviewers consistently favor one style, the model may become bland, over-polished, or reluctant to show uncertainty in a useful way. Human preference is not one thing; it changes by domain, user, task, risk level, and culture. A good training program preserves useful variation instead of sanding every response into the same shape.

⚠️

Proxy discipline. A reward score is useful only when it is continuously checked against real human judgment and real task outcomes.

Optimizing a proxy invites reward hacking, preference collapse, over-refusal, and overfit to a favored style.

Evaluation is not optional

Preference training changes behavior, so it needs evaluation before and after training. The first question is whether the model improved on the tasks that matter. The second is whether it got worse somewhere else. A model can become more helpful in customer support while becoming less precise in legal drafting, or safer on obvious abuse prompts while more evasive on normal technical questions.

Good evaluation mixes automated tests, expert review, adversarial testing, and live monitoring. Automated tests are fast and repeatable, but they miss nuance. Expert review is slower, but it catches domain-specific errors. Adversarial testing looks for edge cases, manipulation, and policy failures. Live monitoring checks whether behavior holds up under real use.

The evaluation set must be protected. If the same prompts are used again and again during training decisions, the model can become tuned to the test rather than the world. Teams should keep holdout sets, rotate fresh examples, and track regressions by category, not just by one overall score.

Measure helpfulness, correctness, safety, tone, and instruction following separately.
Use domain experts when mistakes have legal, medical, financial, security, or operational consequences.
Keep a protected holdout set that is not used to steer daily training.
Track regressions, not only average improvement.

Good evaluation stacks fast automated tests, expert review, adversarial probing, and live monitoring.

The real decisions for leaders and practitioners

The hardest decisions are often not about the optimizer. They are about what the organization is trying to teach the model. Who defines preferred behavior. Which users and domains are represented in the feedback. Which mistakes are unacceptable. How much variation is allowed. Who can approve a change that alters model behavior.

Teams also choose where to spend effort. If the base model is weak for a domain, supervised fine-tuning or retrieval may help more than preference optimization. If the model is already strong but has tone, refusal, or format problems, preference data may be the right lever. If the product depends on current private information, retrieval and tool use may matter more than further training.

Build-versus-buy decisions should include data sensitivity, audit needs, talent, cost, and speed. Managed platforms such as Microsoft Foundry can reduce infrastructure work, but they do not remove the need to define evaluation criteria, govern feedback data, and monitor behavior after release. Preference optimization is a product and governance practice as much as a modeling technique.

A practical mental model

Think of RLHF and preference optimization as ways to teach taste under constraints. The model already knows many ways to answer. Preference training nudges it toward the kinds of answers people choose, while trying not to destroy the broad capability learned during pretraining.

That makes the method powerful but bounded. It can improve style, instruction following, refusal behavior, and alignment with a defined product policy. It cannot guarantee truth, replace domain grounding, or solve unclear business requirements. It learns from the preferences you collect, the rubrics you write, and the tradeoffs you accept.

The best implementations are modest in their claims and rigorous in their checks. They treat human feedback as data with provenance, bias, cost, and drift. They compare models on real tasks. They watch for regressions. And they remember that a model optimized to be preferred is not automatically a model optimized to be right.

✅

Core idea. Preference optimization is most useful when the desired behavior is clear enough for people to compare and important enough to measure.

Key takeaways

RLHF means reinforcement learning from human feedback: people express preferences, and those preferences shape model behavior.
Classic RLHF trains a reward model from human comparisons, then optimizes the language model against that reward.
Direct preference methods such as direct preference optimization can use chosen and rejected responses without a separate reward model.
The quality of preference data and rubrics matters more than the elegance of the optimizer.
Reward hacking, style collapse, and hidden regressions are practical risks that require continuous evaluation.
Leaders must decide what preferred behavior means, who defines it, which risks matter, and how changes will be governed.

Glossary

Reinforcement learning from human feedback: A training approach that uses human judgments about model outputs to create a reward signal for improving future behavior.
Preference optimization: A family of methods that trains a model to favor responses people prefer over responses they reject.
Reward model: A model trained to predict how strongly people would prefer a response in a given context.
Supervised fine-tuning: Training a model on examples of desired inputs and outputs so it learns to imitate those examples.
Direct preference optimization: A method that updates a model directly from chosen and rejected responses without training a separate reward model.
Reference model: A fixed model used as an anchor so preference training does not drift too far from the original behavior.
Reward hacking: A failure mode where a model learns to exploit a scoring system rather than genuinely improve the intended outcome.
Holdout set: A protected group of test examples kept out of training so teams can measure whether a model generalizes.