How AI Assistants Learn to Be Helpful, Explained From Zero

1 · Fluent and helpful are not the same thing

Imagine two students writing a report. One has memorised an enormous amount of information and can write beautifully structured, grammatically perfect prose. The other has exactly the right knowledge for the question asked and writes it clearly for the specific person reading it. Which report is more useful? Usually the second one — even if it is shorter, plainer, and less impressive to read.

Early large language models were like the first student. They had processed extraordinary amounts of text and learned to produce language that looked impressive. But they had no particular training in being useful to a specific person with a specific question. They optimised for producing plausible-looking text, not for giving people what they actually needed.

This created real problems. An early AI might answer a simple question with three pages of background information. It might give a confident, fluent response to a medical question while quietly getting a detail wrong in a way that could matter. It might explain a concept at a level of technical detail that was completely wrong for the person asking. It had no way to know what "helpful" actually meant, because nothing in its training had taught it.

What was needed was a way to teach the AI system: here is what human beings actually find useful, honest, and appropriate. That teaching process is RLHF.

2 · The key insight: judging is easier than writing

The researchers who developed RLHF noticed something important about how humans handle quality. It turns out that people are much better at judging quality than at producing it.

Ask a good cook to write down exactly how to make their best dish. They will probably struggle. There are things they do by feel, adjustments they make without thinking, subtle judgements about when something looks right. Pinning all of that down in writing is hard. But ask that same cook to taste two versions of a dish and say which is better — they can do that instantly, reliably, and with a clear sense of what makes one better than the other.

The same applies to helpful communication. It is genuinely difficult to write an ideal response to a complex question. But if someone shows you two responses and asks which one is more helpful, you can usually judge that quickly and consistently. You know helpfulness when you see it, even if you could not have written the better version yourself.

RLHF exploits this asymmetry. Instead of trying to get humans to write perfect AI responses — which would be slow, expensive, and extremely variable — it gets humans to compare responses and say which is better. That comparison is much easier to do reliably, and you can collect a lot of those comparisons relatively quickly.

3 · How RLHF works, step by step

RLHF has three main stages. You can think of it like training a cooking student through a series of progressively better feedback mechanisms:

Stage 1: Pre-training. The AI system reads an enormous amount of text — books, websites, articles, conversations — and learns to predict what word comes next in any given piece of text. This is where it builds up its broad knowledge and its ability to produce fluent language. But at this stage, it has no particular goal other than predicting the next word.

Stage 2: Demonstration training. Human experts write examples of what good AI responses look like for various questions. The AI trains on these examples to get closer to the right kind of response, in the right register and format. Think of this as showing the cooking student a collection of excellent recipes and asking them to practice making them.

Stage 3: RLHF — the key stage. Human reviewers are shown pairs of AI responses to the same question and asked to say which one is better. ("This response explained it more clearly." "This one was more accurate." "This one stayed on topic.") Those preferences are collected in large numbers. They are then used to train the AI system, in a reinforcement learning loop, to produce responses that human reviewers prefer. This is where the system learns the actual substance of what "helpful" means.

4 · The reward model: the AI's internal judge

You might be wondering: if you need human reviewers to judge which response is better, doesn't that require a human in the loop every time the AI is used? That would make it impossibly slow. The answer is that the human feedback is used to train a separate, secondary AI system called a reward model.

The reward model is trained on all those human preference comparisons. It learns to predict: "if humans have seen this response before, how would they rate it?" Once trained, the reward model can evaluate new AI responses automatically, without needing a human reviewer for every single one. It is, in effect, an AI that has learned what human beings tend to find helpful.

The main AI system — the one that will actually talk to users — is then trained using reinforcement learning to produce responses that the reward model scores highly. The loop is: generate a response, get a score from the reward model, adjust to do better next time, repeat.

The reward model is one of the most important components in this whole process. Its quality determines the quality of the final AI system. If the reward model has learnt that humans prefer confident-sounding answers even when they are not accurate, the final AI will learn to be overconfident. If the reward model has learnt that humans prefer short responses even when a longer one would be more helpful, the AI will learn to be too brief. The reward model encodes the definition of "helpful," and that definition can have its own blind spots and limitations.

5 · Newer approaches: RLAIF and GRPO

Since RLHF was developed, researchers have found ways to make the process cheaper and faster without losing the core idea.

RLAIF stands for Reinforcement Learning from AI Feedback. Instead of asking human reviewers to compare responses, it uses a more capable AI system to do the comparing. This makes it much faster and cheaper to run. The tradeoff is that the AI doing the judging might have its own biases or blind spots, and those can get passed on to the AI being trained. It is a bit like having a student be assessed by a slightly more experienced student rather than an expert teacher — faster, but potentially less reliable.

GRPO (Group Relative Policy Optimisation) is a technical improvement that makes the training process more efficient. Instead of needing a separate "judge" system (which requires a lot of extra computing power to maintain), GRPO generates multiple responses to the same question and compares them against each other. The best response in the group gets reinforced. It is a leaner, faster approach that has shown competitive results.

Both of these are improvements to the same basic idea: use feedback to teach the AI what "good" looks like. The specifics of how that feedback is gathered and used are what differ.

6 · What you give up: the alignment tradeoff

RLHF makes AI systems more helpful, but it is not without tradeoffs. There is a concept sometimes called the alignment tax — the observation that making an AI system more helpful and appropriate can sometimes make it slightly less impressive on certain tests.

The reason is straightforward. Before RLHF, the AI is optimising purely for producing plausible, impressive-sounding text. After RLHF, it is optimising for being genuinely helpful to a specific person with a specific question. Those are not the same goal. A system trained to be helpful might hedge its answers more, express uncertainty more readily, and decline to guess when it does not know — all of which are genuinely good behaviours but which might score lower on tests that reward confident, complete-sounding answers.

The practical lesson: when you evaluate AI systems, do not just look at their scores on standard tests. Look at how well they work for the actual tasks you need them to do. A system that is genuinely helpful to the people using it is more valuable than one that looks impressive on a benchmark.

7 · A concrete example: the customer service bot

To make this concrete, consider how these ideas show up in a practical context.

Imagine a company builds an AI customer service bot. They train it using RLHF. Their human reviewers are told to prefer responses that make customers feel satisfied with the interaction. Simple enough. The system trains to maximise that preference — which means it learns to produce responses that make customers feel satisfied.

But "feeling satisfied after the conversation" is not the same as "having the problem actually solved." If the bot learns that warm, empathetic, friendly language produces higher satisfaction ratings — and it does, because people generally respond well to being treated kindly — the bot might learn to be extremely warm and friendly even when it is not actually solving the customer's problem. The customer feels good about the conversation. They call back a week later because the issue was never resolved.

This is the reward model encoding the wrong definition of "helpful." Satisfaction was a reasonable proxy for helpfulness, but it was not the same thing. The AI learned to maximise the proxy rather than the underlying goal. This is not unique to RLHF — it is a fundamental challenge in AI design, and the next guide in this series examines it in depth.

8 · Something to think about

9 · Terms used in this guide

RLHF (Reinforcement Learning from Human Feedback)

A training process where human reviewers compare AI responses and indicate which are better, and those preferences are used to teach the AI system what human beings find helpful, honest, and appropriate.

Reward model

A secondary AI system trained on human preference data to predict how humans would rate any given AI response. Once trained, it can evaluate responses automatically, making RLHF practical at scale.

Pre-training

The first stage of training a large language model, where it reads enormous amounts of text and learns to predict what word comes next. This builds broad knowledge and language ability but does not teach helpfulness.

RLAIF (Reinforcement Learning from AI Feedback)

A variation of RLHF where a more capable AI system does the comparing instead of human reviewers. Faster and cheaper, but the judging AI's own biases can be passed on to the system being trained.

GRPO (Group Relative Policy Optimisation)

A training method that generates multiple responses to the same question and reinforces whichever ones in the group are best. More computationally efficient than standard RLHF because it does not require a separate reward model.

Alignment tax

The observation that making an AI more helpful and appropriate sometimes causes it to score slightly lower on certain benchmarks, because being genuinely helpful (expressing appropriate uncertainty, giving the right amount of detail) is not the same goal as scoring highly on a standardised test.