Probes as Detectors: Classifying Jailbreak, Refusal, and Unsafe States From Activations

It is by now well established that a model's internal activations carry readable signals about its state. Alain and Bengio showed that a simple classifier on intermediate activations can reveal what a layer represents; Azaria and Mitchell showed that a classifier on hidden activations can tell whether a statement is true with useful accuracy; Burns and colleagues showed that some latent knowledge can be read without labels at all. For a defender, the exciting implication is that the states that matter for security — being jailbroken, being about to comply with a harmful request, representing unsafe content — might be readable from activations before they reach the output. But a readable state is not yet a trustworthy detector. Turning a probe that separates two conditions in a controlled study into a control you can run in production is a data-science problem: you must choose where to read, validate the read against a shifting distribution, report the false-negative rate a security team actually cares about, and reason about base rates before an alarm means anything. This paper is about that gap — how to build a probe detector for security states, and how to know whether to believe it.

Which States a Defender Wants to Read

Before building a detector, name the target precisely, because a probe is only as good as the state it is trained to separate. Three states are especially useful to a defender. The jailbreak state is whether the model has been pushed out of its aligned behavior by an adversarial prompt — a condition that often precedes an unsafe output. The refusal-or-compliance state is whether the model is about to decline a request or fulfill it, which for a harmful request is exactly the fork a defender wants to catch. The unsafe-content state is whether the activations represent harmful material being produced, independent of the surface phrasing.

Each of these is a candidate for an activation probe because each is a semantic condition the model must represent internally to act on. If the model behaves differently when jailbroken, that difference is computed somewhere in its activations, and a probe can in principle read it. The same logic applies to the compliance decision and to the representation of unsafe content.

The security value differs by state and by timing. A jailbreak-state detector is an early warning; a compliance detector catches the decision at the moment it is made; an unsafe-content detector is closer to the output but still ahead of it. A defender chooses which to build based on where in the process an early, reliable signal is most useful.

Building a Probe Detector

The construction follows a standard supervised pipeline adapted to activations. Collect examples of the target state and its absence — prompts that jailbreak the model and prompts that do not, for instance — and record the model's activations at a chosen layer for each. Train a simple classifier, typically linear following Alain and Bengio, to separate the two activation sets. Validate it on held-out examples, and only then consider deploying it as a runtime scorer that reads activations on each forward pass and outputs a probability that the state is present.

Simplicity is a feature, not a limitation. A linear probe is cheap to run, easy to interpret, and its accuracy is a clean measure of how linearly separable the state is at the chosen layer; if a linear probe cannot read the state, that itself is informative. Complex probes can overfit the training distribution and give a false impression of reliability, so a defender generally starts linear and adds complexity only with evidence that it generalizes.

The pipeline's discipline is where the security value is won or lost. A probe trained and tested on the same narrow set of jailbreaks will look excellent and fail on a new one; the sections that follow are about the choices — layer, supervision, validation — that determine whether the detector survives contact with a real, shifting adversary.

Where to Read: Layer and Representation Choice

A model represents different things at different depths, so where a probe reads matters as much as how it is trained. Alain and Bengio's core observation is that intermediate layers carry different, increasingly abstract information, and a security state may be linearly separable at one depth and entangled at another. Early layers tend to carry surface features of the input; middle layers often carry the abstract concepts a defender wants; later layers are shaped toward the output and may already reflect the decision rather than the state that produced it.

The practical method is to probe across layers and choose the depth where the target state is most separable and most stable. Separability tells you the state is readable there; stability across inputs tells you the read will hold in deployment. A detector that reads at a well-chosen middle layer often generalizes better than one forced to read at a layer where the state is only weakly represented.

Layer choice also interacts with timing. Reading earlier gives more warning but a fuzzier signal; reading later gives a sharper signal but less lead time before the output. A defender picks the depth that balances the reliability of the read against how early the warning is needed, and records that choice as part of the detector's specification.

Supervised Versus Unsupervised Detectors

When labeled examples of the target state are available, a supervised probe is the direct route: Azaria and Mitchell's truthfulness classifier is a canonical example, trained on activations labeled true or false and achieving useful accuracy. Supervised probes are as good as their labels and their coverage, so they excel when the state is easy to label and the training set spans the conditions the detector will face.

When labels are scarce, expensive, or the state is hard to define by example, unsupervised methods become attractive. Burns and colleagues discovered latent knowledge in activations without labels by exploiting logical consistency — the requirement that a statement and its negation cannot both be true. For a defender this matters because some security states are hard to label exhaustively, and a method that reads a state from internal structure rather than curated examples can generalize beyond the specific cases a labeler thought of.

The trade-off is coverage versus assumption. Supervised probes are precise about the state they were taught but blind to states outside their labels; unsupervised probes make a structural assumption that may or may not hold for the target state but can cover cases no labeler enumerated. A mature detection strategy often uses both, and validates each against the other's blind spots.

Validating a Detector Like a Data Scientist

A probe becomes a detector only after validation that treats it as the classifier it is. The first measure is separability and held-out accuracy: how well the probe distinguishes the state on data it did not train on. The second, and more security-critical, is behavior under distribution shift: accuracy on jailbreaks, phrasings, or content the probe has not seen, because a real adversary will not reuse the training distribution. The third is the confusion structure — precision and recall — and within it the false-negative rate, the fraction of genuine unsafe states the detector misses, which is the number a security team must own.

False negatives deserve primacy because their cost is asymmetric. A false positive wastes a review; a false negative lets an unsafe state through undetected, which is the failure the detector exists to prevent. A detector reported only by overall accuracy hides its false-negative rate, so the validation protocol must surface it explicitly and set the alarm threshold to trade false positives for the false-negative rate the deployment can tolerate.

The protocol, stated as a checklist, is what separates a paper result from a deployable control: measure held-out separability, measure accuracy under deliberate distribution shift, report the false-negative rate at the operating threshold, and calibrate against the base rate the next section warns about. Only a detector that has passed all four should be trusted as a runtime trip-wire.

The Base-Rate Trap

A detector's accuracy on a balanced test set can badly mislead when the state it detects is rare in deployment, and this is the single most common way probe detectors disappoint. If unsafe states are a small fraction of real traffic, even a detector with high accuracy will produce far more false alarms than true ones, because the many safe cases each contribute a small false-positive chance that swamps the few true positives. Overall accuracy, measured on a balanced set, says nothing about this.

The consequence for a defender is that a detector cannot be evaluated in isolation from its base rate. The same probe that looks excellent in a balanced study can be nearly useless as an alarm in production, not because it got worse, but because the rarity of the event makes its positive predictions mostly wrong. A defender must reason about the deployment base rate explicitly and set thresholds and downstream handling accordingly — for example, by using the detector to prioritize review rather than to auto-block, when the base rate is low.

This is a data-science point, not an interpretability one, but it is where interpretability-based detectors most often fail in practice. The base rate belongs in the validation protocol precisely because a probe that ignores it will generate alarm fatigue and be switched off, taking its genuine detections with it.

Deploying the Detector as a Runtime Trip-Wire

Deployed, a probe detector reads activations on each forward pass, scores the target state, and — when the score crosses a base-rate-calibrated threshold — escalates: it can flag for human review, tighten downstream action gating, or halt the agent, depending on the state and the tolerance for false positives. It is a trip-wire, positioned earlier than an output filter and reading the computation directly, and its job is to raise suspicion in time to act.

The trip-wire must be integrated as one layer, not a verdict. Because the detector has a known false-negative rate, it will miss some unsafe states, so it must sit alongside input controls, output filters, and action gating that bound what a missed detection can do. Its value is precisely that its failures do not coincide with those other layers' failures — an unsafe state that evades an output filter may still trip the internal detector, and vice versa.

Operationally, the detector's score should travel with its assurance — its measured false-negative rate and base-rate calibration — so downstream components weight it correctly. A trip-wire whose reliability is known is a useful control; a trip-wire trusted as certainty is a liability, because its inevitable misses will be treated as all-clears.

What This Means for the AI-Agent Stack

For an agent, a probe detector is an internal trip-wire that watches the reasoning for security states the input and output layers cannot see directly. An agent that has been jailbroken by a hostile instruction, or that is about to comply with a harmful sub-goal, may reveal that state in its activations a step before it acts, and a well-validated detector can escalate or halt in that window. This is the internal complement to the sandboxing and action-gating that bound an agent's behavior from the outside.

The discipline that makes this safe is the validation protocol and the base-rate honesty that go with it. An agent's internal detector is worth deploying when its false-negative rate is known and its threshold is calibrated to the real base rate of the unsafe state, and it is dangerous when it is trusted as a guarantee, because its misses will then be read as safety. As one layer among several, with failures that do not align with the others, it strengthens defense in depth; as a single point of trust, it weakens it.

The single idea to carry is that a probe can read a security state, but only a validated probe — held-out, shift-tested, false-negative-reported, and base-rate-calibrated — is a detector you can deploy, and even then only as one honest layer, never as the verdict.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE detector evaluation. Reports FN rate and base-rate view.
heldout_acc = eval(probe, heldout)
shift_acc   = eval(probe, shifted_set)      # unseen jailbreaks/phrasings
fn_rate     = false_negatives(probe, heldout) / positives(heldout)
# base-rate view: expected precision at deployment prevalence
prod_precision = expected_precision(probe, base_rate=deployment_prevalence)
report(heldout_acc, shift_acc, fn_rate, prod_precision)
trust = (shift_acc ok) and (fn_rate <= tolerance) and (prod_precision usable)

Glossary

Probe detector

A classifier trained on activations to score a security-relevant state, deployed as a runtime signal.

Jailbreak state

The internal condition of a model pushed out of its aligned behavior by an adversarial prompt.

False-negative rate

The fraction of genuine unsafe states a detector misses at its operating threshold — the security-critical metric.

Distribution shift

The gap between the data a probe was trained on and the unseen inputs an adversary will present in deployment.

Base rate

How common the target state is in real traffic, which determines what a positive alarm actually means.

Detector Validation Protocol

The four checks — held-out separability, shift accuracy, false-negative rate, base-rate calibration — required before trusting a probe as a control.

References

1. Alain & Bengio, Understanding intermediate layers using linear classifier probes (arXiv 1610.01644, 2016)
2. Azaria & Mitchell, The Internal State of an LLM Knows When It's Lying (arXiv 2304.13734, EMNLP Findings 2023)
3. Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (arXiv 2212.03827, ICLR 2023)
4. Cunningham, Ewart, Riggs, Huben & Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models (arXiv 2309.08600, ICLR 2024)
5. Zou, Phan, Chen et al., Representation Engineering: A Top-Down Approach to AI Transparency (arXiv 2310.01405, 2023)
6. NIST AI Risk Management Framework (AI RMF 1.0)
7. OWASP Top 10 for LLM Applications
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)