Interpretability's Limits as a Control: False Negatives, Goodharting the Probe, and Layered Defense

A control is trusted exactly as far as its failure modes allow, so the honest way to close an examination of interpretability as a security control is not to celebrate what it can read but to state precisely what it cannot guarantee. Three limits recur in any use of interpretability as a security control. Every internal detector has an irreducible false-negative rate, so it will miss some unsafe states no matter how well built. Every internal signal is subject to a Goodhart dynamic, degrading the moment it becomes an optimization target for training or for an adaptive model. And every monitor faces a coverage-and-cost gap, because watching the whole computation on all traffic is unaffordable, leaving parts of the process unwatched. None of these limits is a reason to abandon interpretability — the earlier, richer signal it provides is real and valuable — but together they decide how it must be used. This paper turns the three limits into a single doctrine, whose load-bearing rule is that interpretability may be used to raise the alarm and never to clear the model, and whose conclusion is that internal monitoring belongs as one failure-uncorrelated layer in defense in depth, never as the last line.

Why Limits, Not Capabilities, Are the Point

The capabilities of interpretability as a control are, by this point, established: a probe can read a security-relevant state earlier than the output and more reliably than a single neuron, and that is genuinely useful. But a security control is not judged by its best case; it is judged by how it fails, because an adversary lives in the failure modes. A detector that is right most of the time but fails silently in the cases that matter most is worse than useless if it is trusted as a guarantee, because it converts a genuine risk into false confidence.

This inverts the usual framing. Rather than asking what interpretability can detect, the doctrine asks what its detections are worth given how they can be wrong, and derives a usage rule from the answer. The three limits below are not caveats appended to a capability; they are the substance from which correct usage is derived.

The payoff of leading with limits is a control that is trustworthy precisely because its authority is bounded to what it can support. A team that knows exactly what its internal monitor may and may not be trusted to do can rely on it without being betrayed by it, which is the only kind of reliance worth having for a security control.

Limit One: The Irreducible False Negative

Every internal detector is a classifier with a false-negative rate — a fraction of genuine unsafe states it misses at its operating threshold — and that rate cannot be driven to zero. Lowering the threshold to catch more true positives raises false positives until the detector is unusable; raising it to reduce noise lets more unsafe states through. There is no setting at which a probe both alarms reliably and never misses, because the underlying signal is imperfect and, on rare states, base rates make even a strong detector's positives mostly wrong.

The consequence is categorical: an internal detector can never constitute a proof of safety. A clean pass means 'this detector, at this threshold, did not fire', which is consistent with both a safe state and an unsafe state the detector missed. The absence of an alarm is not the presence of safety, and any process that treats a quiet probe as a clearance has misread evidence as a guarantee.

This limit is not a defect to be engineered away; it is a property of using an imperfect signal against a real distribution. The correct response is not a better threshold but a usage rule that never asks the detector to certify absence — a rule this paper builds toward and states below.

Limit Two: Goodharting the Probe

The second limit is that an internal signal degrades once it becomes a target. If a probe's output is used as an optimization objective in training, or if an adaptive model has any incentive to reduce it, the internal correlate the probe reads can move, weaken, or relocate to representations the probe does not cover. This is the Goodhart dynamic: a measure that becomes a target ceases to be a good measure, and a probe optimized against becomes a worse probe.

The dangerous specific case is training against the probe to suppress a behavior, which can reduce the readability of that behavior without reducing the behavior itself — producing a model that looks clean to the monitor while being no safer, and thereby manufacturing false confidence. A monitor is most trustworthy on a model that has not been optimized to defeat it, and least trustworthy exactly where the incentive to evade is strongest, which is the highest-risk case.

This limit bounds not just accuracy but the conditions of validity. A detector's assurance must be stated under an explicit assumption about how much optimization pressure the monitored model has been under, and a probe relied upon should not be one the system has been trained hard against. The doctrine encodes this as a prohibition on using a monitor you have optimized against as if it were still reliable.

Limit Three: Coverage and Cost Gaps

The third limit is practical but no less binding: watching the whole computation on all traffic is unaffordable, so a real monitor covers only some layers, tokens, and requests, and the uncovered space is a gap. Because monitoring adds compute and latency on every watched pass, a defender must ration coverage, and every rationing decision leaves moments the monitor is not watching. A detector that is perfect where it looks still sees nothing where it does not.

Gaps are worse than missed coverage because they are exploitable. An adversary who can infer the sampling schedule, the watched tokens, or the escalation trigger can time an attack into the unwatched space, turning a cost-saving decision into a security hole. Predictable gaps are a map of where not to be seen, so coverage decisions are part of the threat model, not an implementation detail.

This limit means an internal monitor's real coverage is always partial and always has an adversarial dimension. The response is to make gaps unpredictable where feasible and, more robustly, to ensure the gaps are covered by controls whose failures do not coincide with the monitor's — which is the layered-defense principle the doctrine builds on.

The Three Limits Compose Into One Rule

The three limits point to a single usage rule, and it is the load-bearing contribution of this paper. Because a detector can miss (limit one), can be evaded under optimization (limit two), and cannot watch everything (limit three), the one thing it can never do is establish that a model or a computation is safe. But every one of those limits leaves intact the detector's ability to raise suspicion: a positive detection, even from an imperfect, partially-covering, non-adaptive-assumption probe, is real evidence that something is wrong and warrants escalation.

So the rule is asymmetric: use interpretability to alarm, never to clear. A positive is informative and should trigger action — human review, tighter gating, a halt. A negative is not a clearance; it is the absence of one particular signal, consistent with an unsafe state the detector missed, evaded, or never watched. Any workflow that lets a quiet probe authorize an action, relax a control, or certify a model has inverted the rule and will be betrayed by exactly the cases the limits describe.

This asymmetry mirrors the one-sided character of the whole field: internal signals are evidence in the direction of alarm and never in the direction of safety. Encoding it as a rule is what lets a team extract interpretability's genuine value — earlier detection of some unsafe states — without granting it the authority to declare safety that its limits forbid.

Where Interpretability Belongs in Layered Defense

Given a control that can alarm but not clear, its correct place is as one layer in defense in depth, chosen precisely because its failures do not line up with the others. Input controls filter hostile prompts; output controls filter unsafe responses; action controls bound what an agent can do; and internal monitoring watches the computation in between. An attack that evades the input filter may still trip the internal monitor; a state the internal monitor misses may still be caught by the output filter or contained by the action gate. The layers cover each other because they fail differently.

This is why interpretability adds real value despite its limits: it is a monitoring surface that no other layer provides, and its distinct failure profile is exactly what defense in depth needs. A stack that adds internal monitoring gains coverage of the reasoning that input and output controls cannot see, and loses nothing, provided the internal monitor is treated as one layer among several rather than as a replacement for any of them.

The load-bearing controls remain the ones that do not depend on reading honest internals — sandboxing, action gating, output filtering — because those hold even against a model that has defeated the probe. Interpretability sits on top of them as an early-warning layer, valuable for catching some unsafe states sooner, never as the foundation on which safety rests.

The Doctrine, Stated

Collapsed to its tenets, the doctrine for using interpretability as a security control is five statements. Treat every detector as evidence with a recorded false-negative rate, never as a proof of safety. Never rely on a probe you have optimized against, and state each detector's assurance under an explicit adaptive assumption. Budget coverage deliberately and name the gaps, making them unpredictable or covering them elsewhere. Use interpretability to alarm and never to clear. And place it as one failure-uncorrelated layer in defense in depth, on top of controls that do not depend on honest internals.

These tenets are not independent preferences; each descends from one of the three limits, and together they form a coherent stance that a team can adopt as policy. A detector deployed under this doctrine is used for its genuine value — earlier detection of some unsafe states — while being refused the authority its limits forbid, which is precisely the discipline that keeps a promising technique from becoming a liability.

The doctrine is also portable across the whole space of internal-signal methods, from probes to sparse-autoencoder features to representation reading, because it is built on limits that all of them share. Whatever the method, it is evidence not proof, degradable under optimization, partial in coverage, usable to alarm not to clear, and best deployed as one layer — and a team that holds to those tenets can adopt new interpretability methods as they mature without re-litigating how to trust them.

What This Means for the AI-Agent Stack

For an agent, this doctrine makes internal monitoring a valuable but strictly bounded layer. An agent that has been jailbroken or is forming a deceptive intent may reveal it internally before acting, and an internal monitor that alarms on that state can escalate or halt earlier than any output check — a real gain. But the same monitor's clean pass can never authorize a consequential action, because the false-negative, Goodhart, and coverage limits all apply, and an agent that could be optimizing against the probe is exactly the case where a quiet monitor is least trustworthy.

So the agent stack uses interpretability as an early-warning tripwire layered over controls that hold without it: sandboxing that bounds reach, action gating that stops irreversible effects, and output filtering, each of which contains the damage of a state the internal monitor missed. The internal monitor makes the stack catch more unsafe states sooner; the other layers ensure that the ones it misses are still survivable. That composition — alarm from the inside, containment from the outside — is the mature way to use interpretability for agents.

The single idea to carry is that reading the machine buys a defender genuine, earlier evidence and never a guarantee, so the discipline that turns it into safety is to use it to raise the alarm, never to clear the model, and to place it as one honest layer in a defense that does not depend on it.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

on internal_signal:
  if positive: escalate([human_review, tighten_gate, halt])  # alarm
  if negative: do_nothing                                    # NOT a clearance
forbidden:
  - negative_signal authorizes action
  - negative_signal relaxes a control
  - negative_signal certifies model safe
assurance: record(false_negative_rate, adaptive_assumption, coverage_gaps)

Glossary

False-negative rate

The fraction of genuine unsafe states a detector misses at its operating threshold; it cannot be driven to zero.

Goodhart dynamic

The failure in which a signal ceases to be a reliable measure once it becomes an optimization target.

Coverage gap

The layers, tokens, or requests a monitor does not watch under its cost budget, which an adversary can exploit.

Alarm, never clear

The rule that internal signals may raise suspicion and trigger action but may never certify safety.

Failure-uncorrelated layer

A control whose failure modes do not coincide with those of the other layers, so it covers their gaps.

Defense in depth

Composing controls with non-aligned failures so an attacker must defeat every layer while the defender needs one to hold.

References

1. Alain & Bengio, Understanding intermediate layers using linear classifier probes (arXiv 1610.01644, 2016)
2. Cunningham, Ewart, Riggs, Huben & Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models (arXiv 2309.08600, ICLR 2024)
3. Zou, Phan, Chen et al., Representation Engineering: A Top-Down Approach to AI Transparency (arXiv 2310.01405, 2023)
4. Azaria & Mitchell, The Internal State of an LLM Knows When It's Lying (arXiv 2304.13734, EMNLP Findings 2023)
5. Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (arXiv 2212.03827, ICLR 2023)
6. NIST AI Risk Management Framework (AI RMF 1.0)
7. OWASP Top 10 for LLM Applications
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)