RL for Reasoning & Code — Reasoning RL & Verifiable Rewards

A language model begins as a very good next-word predictor. That is already powerful, but it is not the same as being careful, persistent, or correct. Reinforcement learning changes the question from “What text is likely?” to “What action leads to a better outcome?” For reasoning and code, that shift matters: a model can be rewarded for solving a math problem, passing tests, finding a bug, or explaining a decision in a way that survives inspection.

Start with the simple loop

Reinforcement learning, often shortened to RL, is a way to train a system by letting it try actions and then giving it feedback. The system is called an agent. The situation it acts in is called the environment. The feedback is called a reward. Over many attempts, the agent learns a policy, which is a rule for choosing actions that tend to earn higher rewards.

In a game, the idea is easy to see. The agent makes a move, the game changes, and the agent eventually wins or loses. In language, the moves are tokens, tool calls, plans, edits, or answers. The environment may be a prompt, a compiler, a unit test suite, a proof checker, a human reviewer, or another model acting as a judge.

The key difference from ordinary supervised learning is that RL does not need to show the model the exact right next word at every step. It can reward a final outcome. That makes it useful when there are many possible paths to a good answer, which is exactly what happens in reasoning and programming.

An agent acts, the environment responds, a reward arrives, and the policy improves over many attempts.

Why reasoning and code are different

Reasoning tasks ask a model to connect steps. Code tasks ask a model to produce instructions that another machine can execute. In both cases, the final answer is not just a matter of style. It can be checked. A proof is valid or invalid. A program compiles or fails. A function passes tests or breaks them.

This checkability is valuable because RL needs rewards. If the reward is vague, the model may learn vague behavior. If the reward is grounded in a real test, the model has a clearer signal. A code model can try a patch, run tests, see failures, and revise. A reasoning model can produce an answer, compare it with a known result, and learn which kinds of intermediate work helped.

Still, checkable does not mean easy. A program can pass weak tests while being wrong. A solution can reach the right answer for the wrong reason. A model can learn to format confident-looking explanations without improving the underlying thinking. Good RL for reasoning and code depends as much on reward design as on model size.

⚠️

Tests are not truth. A reward based on tests is only as good as the coverage, realism, and independence of those tests.

From next-token prediction to rewarded behavior

Most modern language models first learn by predicting text from large collections of documents. This stage gives them grammar, facts, patterns, APIs, and common programming idioms. It also teaches imitation. If the training text contains flawed code, shallow explanations, or overconfident answers, the model can imitate those too.

RL is usually applied after this base training. The model already knows how to speak and write code. RL then nudges it toward behaviors that are more useful: checking assumptions, using tools, making smaller edits, asking for missing requirements, or stopping when it cannot justify an answer.

The important idea is that RL changes incentives. A model that is rewarded only for pleasing-sounding answers may become agreeable. A model that is rewarded for passing hidden tests may become more rigorous. A model that is rewarded for safe refusal in dangerous situations may learn boundaries. The training signal shapes the habit.

What counts as a reward

A reward can come from a human, a rule, a tool, another model, or a mixture of all four. Human feedback is useful for qualities such as clarity, helpfulness, and judgment. Rule-based feedback is useful when correctness can be measured automatically. Tool feedback is especially strong for code because compilers, linters, static analyzers, and test runners produce concrete signals.

For reasoning, rewards are harder but still possible. A math answer can be checked against a known result. A symbolic step can be verified by a solver. A multiple-step answer can be judged by whether its conclusion follows from the premises. Some systems reward the final answer only; others also reward useful intermediate steps, such as decomposing a problem or calling the right tool.

The danger is reward hacking, which means finding a way to score well without doing the intended task. A code model might learn to special-case tests instead of fixing the general bug. A reasoning model might learn phrases that a judge tends to like. A safe system must assume the model will optimize the reward literally, not morally.

Human preference rewards capture usefulness but can be subjective and inconsistent.
Compiler and test rewards are concrete but can miss edge cases.
Verifier rewards are strong when the domain has formal rules.
Model-judge rewards scale well but can inherit the judge model's blind spots.
Mixed rewards are often best because each signal covers a different failure mode.

Rewards can come from humans, rules, tools, or another model, each strong for different qualities.

Reasoning is more than showing steps

People often associate reasoning models with long explanations. That can be misleading. A long explanation is an artifact, not the reasoning itself. The real question is whether the model can choose better actions: break down a problem, test a hypothesis, notice a contradiction, use a tool at the right time, and revise when evidence changes.

RL can encourage these behaviors because it can reward outcomes that require persistence. For example, a model working on a programming task may need to inspect files, form a plan, edit one function, run tests, read the failure, and edit again. The reward arrives only after the task succeeds, but the learned policy improves the whole chain.

This is why tool use is central. A model that can call a calculator, search a codebase, run a test, or query documentation is not limited to what it remembers. In Microsoft Foundry, teams can combine models with tools, evaluations, safety controls, and application workflows. The same principle applies anywhere: RL is most useful when the model can act and observe, not merely answer once.

The hard parts: credit, drift, and safety

The first hard part is credit assignment. If a model writes twenty steps and the final answer is wrong, which step caused the failure? If a patch fails one hidden test, was the plan wrong, the edit too narrow, or the test expectation misunderstood? RL learns from delayed feedback, so it needs many attempts and careful evaluation to separate lucky success from reliable skill.

The second hard part is distribution shift, which means the deployment world differs from the training world. A model trained on tidy benchmark problems may struggle with a messy enterprise repository, ambiguous requirements, private libraries, or incomplete tests. Reasoning and code systems must be evaluated on tasks that resemble the work they will actually do.

The third hard part is safety. Code can delete data, expose secrets, introduce vulnerabilities, or violate licenses. Reasoning systems can give false confidence in high-stakes settings. Strong systems use permissions, sandboxing, identity, logging, and human review. Microsoft Entra ID can help control which users and services may access tools and data, but access control is only one layer of the design.

⚠️

Do not reward shortcuts. If the easiest path to reward is unsafe, the model may learn unsafe behavior faster than useful behavior.

Outcome-based learning must handle delayed credit, a messy real world, and real safety risks.

Decisions leaders and practitioners must make

The practical question is not “Should we use RL?” It is “Where would outcome-based learning create value that ordinary prompting or supervised fine-tuning cannot?” If the task has measurable outcomes, repeated attempts, and safe feedback, RL may help. If the task is rare, subjective, or hard to score, better retrieval, clearer instructions, or human workflow design may matter more.

Leaders also need to decide how much autonomy to allow. A model that suggests code is different from a model that opens a pull request, and both are different from a model that deploys to production. Each step requires stronger evaluation, narrower permissions, clearer rollback paths, and better observability.

A useful adoption path is incremental. Start with offline evaluation against realistic tasks. Add tool use in a sandbox. Compare against a non-RL baseline. Measure not only success rate but also cost, latency, security findings, review burden, and failure severity. If you use Microsoft Foundry or Foundry Tools, treat the platform as part of the control plane, not as a substitute for task-specific tests and governance.

Choose tasks with objective, repeatable feedback.
Keep training, evaluation, and production test sets separate.
Use sandboxes for code execution and tool calls.
Require human approval before high-impact actions.
Track failure severity, not just average accuracy.

Adopt outcome-based learning incrementally, adding autonomy only as evaluation and guardrails mature.

How to tell whether it works

A reasoning or code model improved by RL should be judged by behavior, not by the presence of impressive explanations. Good evaluation asks whether it solves new problems, handles edge cases, admits uncertainty, uses tools appropriately, and fails safely. The best tests are hidden from training and close to real work.

For code, useful measures include build success, test pass rate, bug recurrence, security scan results, review time, and the number of unnecessary changes. For reasoning, useful measures include final correctness, calibration, consistency under paraphrase, ability to use evidence, and resistance to misleading prompts.

The strongest sign is not a single benchmark win. It is a pattern: better outcomes on independent tasks, fewer severe failures, stable performance after deployment, and clear explanations of when the system should not be trusted. RL is a training method, not a guarantee. Its value appears only when the reward, the environment, and the evaluation match the work that matters.

✅

Measure the work. A good RL system improves the task you care about, not just the score you happened to optimize.

Key takeaways

Reinforcement learning trains behavior by rewarding outcomes, not by prescribing every next word.
Reasoning and code are good RL candidates because many outcomes can be checked with tests, tools, or verifiers.
Reward design is the center of the problem: weak rewards teach weak or misleading behavior.
Tool use turns a model from a one-shot answerer into an agent that can act, observe, and revise.
Autonomy should increase only with stronger evaluation, permissions, sandboxing, and human review.
Success means better real-world task performance, fewer severe failures, and stable behavior on independent tests.

Glossary

Reinforcement learning: A training method in which a system learns to choose actions by receiving rewards or penalties from its environment.
Agent: The system that chooses actions, such as generating text, editing code, or calling a tool.
Environment: The setting the agent acts in, including prompts, files, tools, tests, users, and feedback.
Reward: A score or signal that tells the agent how desirable an outcome or behavior was.
Policy: The learned rule or strategy an agent uses to choose its next action.
Reward hacking: The failure mode where a system learns to get a high score without doing what people actually intended.
Credit assignment: The problem of deciding which earlier action helped or hurt a later outcome.
Distribution shift: A mismatch between the situations used for training or testing and the situations encountered in real deployment.
Sandbox: A restricted environment where code or tool actions can run with limited permissions and reduced risk.