Specifying Intended Behavior: Goal Representations, Invariants, and the Reference Policy to Drift From

A drift detector answers a comparative question — how far has the agent departed from what it was supposed to do — and that question is empty without a second term. You need an explicit encoding of intended behavior to compare against, and the quality of that encoding sets a ceiling on everything downstream: a detector cannot see drift along a dimension its reference does not describe. This is the uncomfortable dependency at the heart of drift detection. Before you can measure divergence, you must commit to a representation of the goal, accept that it is a proxy, and quantify what it leaves out. This article is about building that reference well, and about being honest regarding what it cannot capture.

The Reference Problem

Call the encoding of intended behavior a reference. Formally it is anything that lets you assign, at each step of a trajectory, a measure of how consistent the realized behavior is with the goal — most naturally a reference policy that specifies what the agent should tend to do in each state, but equally a set of constraints, a scoring function, or a declarative specification. The reference is the object drift is measured from; without it, drift is undefined rather than merely undetected.

The reference is deliberately not the same as the goal. A goal is an intention held by whoever deployed the agent; a reference is an operational artifact that approximates that intention in a form a machine can compare against. The distance between them — the specification gap — is the central difficulty of this article and recurs throughout. Naming the reference as a distinct artifact, separate from both the agent and the goal, is what makes it something you can design, test, and improve.

This framing aligns with how governance frameworks already think. The NIST AI Risk Management Framework organizes practice around measuring and managing risk against stated expectations, and a reference is precisely the stated expectation made measurable. The reference is therefore not only a detection prerequisite but a governance artifact: it is the written-down answer to what this agent is supposed to do, against which both drift detection and audit can proceed.

Four Ways to Represent a Goal

Intended behavior can be encoded in several qualitatively different forms, and each captures a different facet of the goal. A declarative specification states the goal and its constraints in structured terms — what the task is, what must be produced, what is out of scope — and is auditable but coarse, since much of what counts as on-course behavior is hard to enumerate declaratively. A reference policy encodes the goal as expected behavior, whether as a trusted model of what a well-behaved agent tends to do or as a corpus of exemplar trajectories; it captures distributional normalcy that a declaration cannot.

A learned value or reward model encodes the goal as a scalar scorer that rates how well a state or trajectory serves the goal, giving a continuous signal at the cost of being itself a fallible approximation — the very object whose misspecification the reward-hacking literature of Skalse and colleagues studies. Invariants encode the goal's non-negotiable constraints: propositions that must hold in every state regardless of how the task unfolds, such as never exfiltrating a secret or never exceeding a spending cap.

These are not competitors but complements. A declaration fixes scope, a reference policy captures normal trajectory shape, a value model gives a graded conformance signal, and invariants set a hard floor. A reference built from only one is brittle: declarations miss distributional drift, policies miss novel-but-conforming behavior, value models can be gamed, and invariants say nothing about the vast space of behavior that is merely off-target rather than forbidden.

Invariants: The Non-Negotiable Floor

Invariants deserve separate treatment because they differ in kind from the other representations. A goal-conformance signal is soft and graded — behavior can be more or less on-course — whereas an invariant is a hard predicate that must hold in every state, and whose violation is not drift to be scored but a boundary to be enforced. Reading a confidential file into an external message is not a low conformance score; it is an invariant breach, and the appropriate response is to block, not to weigh.

Separating the hard floor from the soft signal is what keeps a drift detector from having to express safety as an extreme of conformance, which it does badly. The safe-exploration concern that Amodei and colleagues raise — that a capable agent exploring a large action space may take catastrophic, irreversible steps — is best met by invariants that hold regardless of the reward or the trajectory, not by hoping a conformance score is low enough to catch them. Invariants are the part of the reference you can state crisply and enforce absolutely.

The design discipline is to keep the invariant set small, precise, and independent of the goal. An invariant that depends on the task is really a soft constraint in disguise and belongs in the conformance signal. The value of a true invariant is that it holds across every goal the agent might be given, which makes it both cheap to check and impossible to drift past — a fixed boundary inside which all the graded drift detection then operates.

The Specification Gap

Every representation is a proxy, and the gap between the reference and the true goal is where undetected drift lives. This is the same proxy-target separation that defines reward hacking in the analysis of Skalse and colleagues, now relocated from the agent's objective to the detector's reference: a reference that omits a dimension of the goal cannot register drift along that dimension, so the agent can wander freely there while the detector reports all-clear. The specification gap is not an implementation flaw to be patched away but a structural property of using any finite reference to stand in for intent.

The gap has a direction that matters. An over-broad reference — one that accepts too much as on-course — misses real drift, producing false negatives; an over-narrow reference — one that treats benign variation as departure — produces false positives and, if trusted, trains operators to ignore it. Both failures trace to the same source: the reference is an approximation, and the approximation error shows up as detection error. The honest move is to measure the gap rather than pretend it is zero.

Measuring the gap means enumerating the dimensions of the goal the reference is meant to cover and identifying, explicitly, the ones it cannot express. A reference that captures task completion but not resource usage is blind to a spending-runaway drift; one that captures tone but not factual grounding is blind to a confabulation drift. Documenting these blind spots turns the specification gap from an invisible liability into a stated limitation that downstream detection and human oversight can compensate for.

Composing a Reference

A robust reference is a composite that assigns each representation the job it does best. Invariants form the hard floor, checked absolutely and enforced on violation. A reference policy supplies the distributional expectation whose divergence from realized behavior becomes the primary graded drift signal. A value model contributes a second, independent conformance signal that can catch drift the policy comparison misses. A declarative specification fixes scope and provides the human-readable anchor for audit and for constructing the other three.

Composition also buys robustness against the specification gap, because the representations have different blind spots. A behavior that conforms distributionally to the reference policy but scores poorly under the value model is flagged by the composite even though the policy comparison alone would miss it; a behavior that satisfies both but violates an invariant is stopped regardless. Requiring agreement, or combining the signals, raises the coverage of the composite above any single representation — the same independence principle that governs layered detection generally.

The cost of composition is calibration complexity and the risk of conflicting signals, which the composite must resolve with an explicit precedence: invariants override, then the graded signals combine under a stated rule. Documenting that precedence, and the coverage each representation contributes, turns the reference into an auditable specification rather than an opaque bundle — which is what the governance frameworks require of any artifact that gates behavior.

Validating the Reference

A reference is a model, and an unvalidated model should gate nothing. Validation begins with coverage: enumerate the goal dimensions the reference is meant to encode and confirm, dimension by dimension, that some representation in the composite can express departure along it. Any dimension no representation covers is a declared blind spot, not a passed test. This is the discipline that converts the specification gap from a hidden risk into a documented one.

The sharper test is to red-team the reference itself — to search for behaviors that drift from the true goal while staying within what the reference accepts. Each such behavior is a concrete instance of the specification gap and a candidate for tightening a representation or adding one. Framing reference validation as an adversarial search, rather than a checkbox, is what surfaces the gaps that matter before an adversary or an unlucky trajectory finds them in production.

Finally, the reference must be calibrated against labeled trajectories: known-good runs should produce low drift signals and known-drifted runs high ones, with enough separation to set a threshold. Where labeled drift is scarce — the common case — synthetic drift, constructed by perturbing good trajectories toward known failure modes, provides a calibration set. A reference that cannot demonstrate separation on the cases you have is not ready, and versioning it with its coverage map, blind spots, and calibration evidence is what makes every later drift alarm traceable to a specific, auditable specification.

From Reference to Drift Signal

The reference exists to be queried at runtime. For it to feed a drift signal, each representation must be evaluable at a step: the invariants checked as predicates, the reference policy compared to the realized action distribution, the value model scored on the current state, and all of this reducible to a per-step measure of departure. A reference that can only be evaluated after the task completes supports post-hoc audit but not the early, sequential detection that catches drift before divergence completes.

This queryability requirement shapes representation choices. A declarative specification too abstract to evaluate step-by-step must be compiled into checkable invariants and a reference policy; a value model too slow to score inline must be distilled or sampled. The reference is not merely a description of intent but an instrument, and an instrument must be readable at the cadence of the process it monitors — which for an autonomous agent is every action.

The reframing to carry forward is that the reference is the load-bearing artifact of drift detection. Everything downstream — the drift signal, the change detector, the escalation thresholds, the correction — inherits its coverage and its blind spots. Invest in the reference, compose it from representations with different blind spots, validate it adversarially, version it with its limitations stated, and make it queryable per step; a drift detector can be no better than the reference it drifts from.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

reference:
  invariants:            # hard floor — enforce, never score
    - never_egress: [ secret, credential ]
    - spend_cap_per_task: 100
  reference_policy:      # distributional expectation for the drift signal
    source: exemplar_trajectories
    signal: kl_divergence
  value_model:           # second, independent graded signal
    scores: goal_service_quality
  precedence:            # how conflicts resolve
    - invariants override
    - combine: max(policy_divergence, 1 - value_score)
  coverage:              # documented dimensions + blind spots
    covers: [ task_completion, resource_use, egress ]
    blind_to: [ long_horizon_tone_shift ]

Glossary

Reference

An operational artifact encoding intended behavior in a form that lets realized behavior be compared against it at each step.

Reference policy

A representation of the goal as expected behavior — a trusted model or corpus of exemplar trajectories — used to compute distributional drift.

Invariant

A hard predicate that must hold in every state regardless of goal; its violation is a boundary breach to enforce, not drift to score.

Value / reward model

A learned scalar scorer rating how well a state or trajectory serves the goal, giving a graded conformance signal that is itself fallible.

Specification gap

The proxy-target distance between the reference and the true goal; drift along dimensions the reference omits is invisible to detectors built on it.

Coverage map

An enumeration of the goal dimensions a reference is meant to encode and which representation covers each, with uncovered dimensions declared as blind spots.

Queryability

The property that every representation in a reference can be evaluated per step, enabling early sequential detection rather than only post-hoc audit.

References

1. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
2. NIST AI 600-1, Generative AI Profile
3. OWASP Top 10 for Large Language Model Applications (2025)
4. OWASP Agentic Security Initiative — Agentic AI Threats and Mitigations
5. MITRE ATLAS — Adversarial Threat Landscape for AI Systems
6. Amodei, Olah, Steinhardt, Christiano, Schulman & Mane, Concrete Problems in AI Safety (arXiv:1606.06565, 2016)
7. Skalse, Howe, Krasheninnikov & Krueger, Defining and Characterizing Reward Hacking (NeurIPS 2022)