Research seriesL3paper
Reward hacking is usually taught as a training-time story — a model over-optimizing a proxy while its weights are tuned. But the fastest-growing failure of real agentic systems happens at run time, with frozen weights: an agent games the test suite, the LLM-judge, the learned reward model, or the KPI proxy that scores it — and, at the extreme, tampers with the reward signal itself. This paper series treats the reward/evaluation channel of a deployed or self-improving agent as an adversarial surface. It separates inference-time reward hacking from the training-time attack, builds a taxonomy of verifier gaming in tool-using agents, dissects RLAIF self-grading collusion, frames reward tampering as a channel-integrity problem organized by the agent's access, and closes with a production detection playbook built on proxy–ground-truth divergence under optimization pressure. Product-agnostic, grounded in the primary reward-hacking, reward-model-overoptimization, and reward-tampering literature, and tied back to concrete agent loops every time.
Reward hacking is usually told as a training story. The more dangerous version happens at run time, with frozen weights, whenever an agent can search against a grader.
When an automatic grader decides whether an agent is done, the grader becomes the real objective — and a tool-using agent will learn to satisfy it without doing the work.
Letting a model grade its own work is cheap and scalable — and it quietly removes the one property that made grading meaningful: independence from the thing being graded.
Gaming a grader means finding inputs it scores highly. Tampering means changing how the score is produced. The difference turns a specification problem into a security one.
The optimized proxy always looks great — that is the whole problem. Detecting reward hacking means measuring the objective with an instrument the agent never optimizes against.