Detecting Drift at Runtime: Trajectory Distance, Reward-Model Monitors, and Statistical Change Detection

A reference tells you what on-course behavior looks like; a runtime detector has to convert the gap between that reference and what the agent is actually doing into a decision, moment by moment, about whether to raise an alarm. Two problems sit inside that sentence. First, how do you measure departure at a single step — as a distance, as a declining score, as something else? Second, how do you decide that a run of departures is a real, persistent shift rather than the ordinary noise of a stochastic agent? The first problem is signal design; the second is change detection. This article treats them in turn, then combines them, and insists throughout that the answer is judged on a curve, not at a point.

Three Families of Drift Signal

Runtime drift detection is built from two composable steps: compute a drift signal that measures per-step departure from the reference, then run a change detector that decides when that signal has shifted persistently. The signal step admits three dominant families, each seeing a different facet of drift. Trajectory distance measures how far the realized behavior has moved from the reference geometrically or distributionally. Reward-model monitors track a learned conformance score and watch for it to decline. Statistical change detection is the decision layer that turns either signal into an alarm.

These families are complementary rather than interchangeable, because they fail on different drifts. A trajectory-distance signal built on action distributions catches a shift in how the agent acts but can miss a drift that keeps the action distribution stable while degrading outcome quality; a reward-model monitor catches declining goal-service but can be gamed by behavior that scores well on the proxy while violating intent — the reward-hacking failure that Skalse and colleagues characterize, now afflicting the monitor. Running them together is what raises coverage above any one.

The organizing frame for the decision layer is sequential change-point detection, the tradition Page initiated: watch a signal over time and declare, as early as possible, that a persistent change has begun. This frame is what supplies both the algorithms and the theory of their limits, and it is why the article treats signal design and change detection as separate concerns joined at a well-defined interface — a scalar drift signal in, an alarm out.

Trajectory Distance

The most direct drift signal measures the distance between realized and reference behavior at each step. When the reference is a policy, a natural choice is a divergence between the realized action distribution and the reference's — for example the Kullback-Leibler divergence, which is large when the agent concentrates probability on actions the reference considers unlikely. This captures drift as a growing mismatch in what the agent tends to do, computed step by step and requiring only that the reference policy be queryable for a distribution.

When behavior is better captured as whole subsequences than as per-step distributions, two alternatives apply. An embedding distance maps the realized trajectory-so-far and the reference's expected path into a vector space and measures their separation, catching structural drift that per-step divergence smears out. A sequence distance, such as an edit distance over the action alphabet, measures how many changes separate the realized action sequence from a reference exemplar, which is interpretable but brittle to benign reordering.

Each metric encodes an assumption about what drift looks like, and choosing one is choosing what you can see. Distributional divergence assumes drift shows up as changed action probabilities; embedding distance assumes it shows up as changed trajectory geometry; sequence distance assumes it shows up as changed ordering. A detector that commits to one metric is blind to drifts that manifest only in another, which is the first reason a single signal is insufficient.

Reward-Model Monitors

A second family scores conformance directly. A learned value or reward model rates how well the current state or the trajectory-so-far serves the goal, and drift appears as a decline in that score over time — not as a single low value, which may just be a hard step, but as a sustained downward trend. Monitoring the trend rather than the instantaneous score is what distinguishes drift from ordinary task difficulty, and it converts the reward model into a time series the change detector can consume.

The reward-model monitor sees something the distance signal can miss: a trajectory that stays distributionally normal while steadily degrading in goal-service, because the agent is doing normal-looking things toward the wrong end. But it inherits the reward model's fallibility. The proxy-target gap that defines reward hacking means the monitor can be high while intent is violated, and an agent whose drift happens to improve the proxy will be invisible to it. This is why the monitor is a second signal, never the only one.

Practically, the monitor must be cheap enough to score inline at the agent's action cadence, which often forces a distilled or sampled value model rather than a full evaluator. The engineering discipline is to keep the monitor independent of the reference policy used for trajectory distance, so that the two signals fail on different drifts; a monitor trained on the same objective as the policy comparison adds little, because their blind spots coincide.

Statistical Change Detection

Whatever the signal, the decision to alarm is a sequential change-point problem: given a stream of drift-signal values that are small under normal operation and larger after drift onset, declare the onset as early as possible. The classical instrument is the cumulative-sum procedure of Page, which accumulates the amount by which the signal exceeds a reference level and alarms when the accumulation crosses a threshold. Its virtue is that transient spikes decay from the accumulator while a persistent shift builds it past the threshold, giving quick detection of real drift with tolerance for noise.

The cumulative-sum statistic has two parameters that expose the fundamental trade-off directly. The slack parameter sets how much excess is ignored as noise; the threshold sets how much accumulated evidence triggers an alarm. Raising the threshold lengthens the average time between false alarms but also lengthens the delay to detect true drift, and lowering it does the reverse. A likelihood-ratio formulation, the sequential probability ratio test, makes the same trade-off explicit as a race between two accumulating hypotheses, normal versus drifted.

Framing detection this way imports the theory that bounds it. The average run length to a false alarm and the expected delay to detect a true change are linked: no detector can shorten one without lengthening the other for a fixed signal quality. This is not a limitation of a particular algorithm but a property of sequential detection, and it means the only way to improve both at once is to improve the signal — to make drift show up larger and sooner relative to noise, which is precisely what combining signal families does.

Combining Signals

Because the signal families have different blind spots, the effective detector fuses them. A trajectory-distance signal, a reward-model monitor, and any auxiliary signal — resource use, invariant proximity — are each fed to a change detector, and their alarms combined under a stated rule: an OR of the detectors maximizes sensitivity, a voting or agreement rule trades some sensitivity for far fewer false alarms. The choice of combination rule is another position on the delay-versus-false-alarm frontier, made once and reported.

Fusion improves the signal quality that the change-detection theory says is the only free lever. A drift that is small in action divergence but large in declining reward becomes detectable sooner in the combined signal than in either alone, shortening delay at a fixed false-alarm rate. The gain is largest when the signals are statistically independent, and smallest when they are correlated, so the design goal is independence of the fused signals — the same principle that governs layered detection throughout agent security.

The cost is calibration: each detector has its own parameters and the combination rule its own, and all must be set against the same evaluation. The discipline that keeps this tractable is to route every signal through one common change-detection interface and to calibrate the whole ensemble on a shared set of labeled and synthetic-drift trajectories, so the reported operating point characterizes the fused detector, not a component in isolation.

Detection Delay Versus False Alarms

Every design choice above resolves to a position on one frontier: expected detection delay against false-alarm rate. A detector tuned for a long average time between false alarms necessarily accepts a longer delay before it catches true drift, and one tuned for fast detection accepts more false alarms. Because drift accumulates, the cost of delay is not merely a late alert but a more diverged trajectory to recover from, which raises the stakes of sitting too far toward the low-false-alarm end.

The frontier is not a nuisance to be tuned away but the object to be characterized and chosen deliberately. The right operating point depends on the cost of a missed drift versus the cost of a false alarm in the specific deployment — a fleet correcting cheaply and reversibly can afford a sensitive setting, while one whose corrections are disruptive must sit more conservatively and lean on making corrections cheaper. Stating the operating point, and the cost model behind it, is what turns detector tuning from folklore into engineering.

This frontier is also why the base-rate problem, treated in full in the next article, is unavoidable: even at a chosen false-alarm rate per decision, the absolute volume of false alarms depends on how often the detector decides and how rare true drift is. The delay-versus-false-alarm curve characterizes a single decision; the base rate scales it to a fleet. A detector cannot be evaluated on the curve alone, but it cannot be evaluated without it either.

Evaluating a Drift Detector

A drift detector is evaluated as a sequential detector, not a classifier. The primary metrics are the average run length to a false alarm and the expected detection delay on true drift, measured on labeled trajectories where the drift onset is known. Where labeled drift is scarce — the usual case — synthetic drift constructed by perturbing known-good trajectories toward the taxonomy's failure modes provides onsets with ground truth, and the detector's delay and false-alarm behavior are measured against them.

Reporting must preserve the frontier. A single detection-rate number hides the operating point and is therefore uninformative; the honest report is the delay-versus-false-alarm curve with the chosen point marked, plus the per-family contribution so consumers can see which signal is carrying the detection. This mirrors the operating-curve discipline of any detector that trades two errors, and it is what lets a reviewer judge whether the detector is tuned appropriately for its deployment's cost model.

Finally, evaluation must probe the blind spots the signal families declared. A detector that scores well on synthetic action-divergence drift but was never tested on outcome-decay drift has an unmeasured gap, and the reward-hacking risk means a monitor should be evaluated specifically against proxy-conforming drift that keeps its score high. Evaluating against the drifts your signals are blind to, rather than only the ones they catch, is what surfaces the coverage holes before production does.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

function step_signal(state, realized, reference, value_model):
    d_kl  = kl_divergence(realized.dist(state), reference.dist(state))
    d_val = max(0, reference.expected_value(state) - value_model.score(state))
    return max(z(d_kl), z(d_val))          # z = normalize on benign; fuse by max

function cusum(signal_stream, k, h):
    g = 0
    for t, d in enumerate(signal_stream):
        g = max(0, g + (d - k))            # slack k ignores noise
        if g > h:                          # threshold h sets delay vs false alarm
            return { "alarm": True, "onset_est": t }
    return { "alarm": False }

Glossary

Drift signal

A per-step scalar measuring departure of realized behavior from the reference, consumed by a change detector.

Trajectory distance

A drift signal measuring geometric or distributional separation between realized and reference behavior, e.g. action-distribution divergence or trajectory-embedding distance.

Reward-model monitor

A drift signal that tracks a learned conformance score over time and flags a sustained decline in goal-service.

Change-point detection

The sequential problem of declaring, as early as possible, that a monitored signal has undergone a persistent shift.

Cumulative-sum (CUSUM)

A change detector that accumulates signal excess over a slack level and alarms when the accumulation crosses a threshold.

Average run length

The expected time between false alarms under normal operation; traded against detection delay.

Detection delay

The expected time between true drift onset and the detector's alarm, minimized subject to a false-alarm bound.

References

1. Page, Continuous Inspection Schemes (Biometrika, 1954)
2. Skalse, Howe, Krasheninnikov & Krueger, Defining and Characterizing Reward Hacking (NeurIPS 2022)
3. Amodei, Olah, Steinhardt, Christiano, Schulman & Mane, Concrete Problems in AI Safety (arXiv:1606.06565, 2016)
4. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
5. NIST AI 600-1, Generative AI Profile
6. OWASP Agentic Security Initiative — Agentic AI Threats and Mitigations
7. MITRE ATLAS — Adversarial Threat Landscape for AI Systems