Reading the Machine: Features, Sparse Autoencoders, and What Interpretable Actually Buys a Defender

A security engineer watching a language model has traditionally had one place to look: the output. But the output is the last thing to change, and by the time an unsafe response is emitted, the decision that produced it has already happened inside the network. Mechanistic interpretability offers a tempting alternative — read the internals, where an unsafe state might be visible before it is expressed. The last few years have made this partially possible: Alain and Bengio showed that simple classifiers trained on intermediate activations can read what a layer represents; Cunningham, Ewart, Riggs, Huben and Sharkey used sparse autoencoders to pull interpretable features out of the tangle of activations; Zou and colleagues framed representation engineering as a way to both monitor and steer high-level phenomena; and Azaria and Mitchell showed that a model's internal state carries a signal of whether a statement is true. Each of these is exciting, but excitement is not a threat model. This paper asks the defender's question instead: given that we can now partially read the machine, what does that legibility actually buy as a security control — and, just as importantly, what does it not?

The Defender's Question

For a defender, the appeal of interpretability is timing and access. The output is a lagging indicator: it reveals an unsafe decision only after the decision is made, and it can be shaped to look benign even when the process that produced it was not. Internals are an earlier and richer signal — the activations that precede the output may reveal that the model is in a jailbroken state, is representing a deceptive intent, or is about to produce a harmful response, before any of that reaches the visible text.

This reframes interpretability from a scientific pursuit into a candidate monitoring control. A control is judged not by how much insight it yields but by how reliably it distinguishes safe from unsafe states, how it behaves against an adversary who knows it is there, and what it costs to run. Those are different questions than the interpretability literature usually asks, and answering them is the point of taking a defensive lens.

The honest framing, established up front, is that reading internals gives a defender a signal, not a certificate. A signal can raise or lower the probability that a state is present; a certificate would prove it. Much of this paper is about keeping that distinction sharp, because conflating the two is how a promising monitoring technique becomes a false sense of security.

What 'Interpretable' Means

Precision in vocabulary prevents most of the confusion in this area. An activation is the numerical state of the network at a point in its computation for a given input. A feature, in the sense used here, is a direction in that activation space that corresponds to a human-meaningful concept — the presence of a topic, a sentiment, a state like 'being asked to do something harmful'. Interpretability is the project of identifying such features and reading their value from activations, so that the internal state becomes partially legible in human terms.

The foundational technique is the probe. Alain and Bengio introduced training a simple linear classifier on the activations of an intermediate layer to test what information that layer linearly represents; if a small classifier can read a concept from a layer, the concept is present there in an accessible form. A probe is thus a reading instrument: it turns an activation into a score for a concept, and its accuracy measures how legibly that concept is represented.

The distinction that matters for a defender is between reading a concept and controlling for it. A probe that reads 'this looks like a jailbreak attempt' with high accuracy is a monitor; it does not, by itself, prevent the jailbreak. Legibility is a prerequisite for a control, but it is not the control, and the rest of this paper keeps that separation in view.

Superposition and Why Neurons Lie

A naive defender might hope to watch a single neuron that means 'harmful' and alarm when it fires. This does not work, and understanding why is central. Cunningham and colleagues describe polysemanticity — individual neurons activate across multiple, semantically distinct contexts — and attribute it to superposition, in which a network represents more features than it has neurons by assigning features to an overcomplete set of directions in activation space rather than to individual neurons. A single neuron therefore participates in many features at once and cleanly signals none of them.

The security consequence is that neuron-level monitoring is unreliable by construction. A neuron that appears to track a concept in one context tracks something else in another, so a monitor built on raw neurons produces both false alarms and missed detections that shift with the input distribution. The signal a defender wants is not in any neuron; it is in a direction that mixes across many neurons.

This is why interpretability for security depends on methods that recover the underlying features from the superposed activations rather than reading neurons directly. The two methods that do so — sparse autoencoders and probes — are the subject of the next sections, and their whole value is that they find the meaningful directions that individual neurons obscure.

Sparse Autoencoders: Unmixing the Signal

A sparse autoencoder is an unsupervised method that learns to reconstruct a model's activations using a large set of features, only a few of which are active for any given input. Cunningham and colleagues showed that the features such autoencoders learn are more interpretable and more monosemantic than directions found by alternative approaches, and that they can pinpoint the features causally responsible for specific model behaviors more finely than earlier decompositions. In effect, the autoencoder unmixes the superposition, turning a tangle of activations into a dictionary of named, sparsely-activating features.

For a defender, the payoff is a vocabulary of concrete features to watch. Instead of an opaque activation vector, monitoring can reference specific learned features — a feature that activates on requests for dangerous instructions, for instance — that are more likely to mean one thing consistently. Because the method is unsupervised and scalable, it can produce a broad feature dictionary without hand-labeling every concept in advance.

The caveats are equally important and set up the ledger later. The features an autoencoder recovers are measured as interpretable by automated methods, not certified as complete or exclusive; some safety-relevant concepts may be split across features or entangled in ways the dictionary does not cleanly separate. A sparse autoencoder gives a defender better raw material for monitoring, not a guaranteed complete map of the model's concepts.

Probes: Reading a Concept From Activations

Where a sparse autoencoder discovers features unsupervised, a probe reads a chosen concept directly. Building on Alain and Bengio's method, a defender trains a classifier on activations labeled for a target state — safe versus unsafe, truthful versus deceptive — and uses it at runtime to score new inputs. Azaria and Mitchell applied exactly this to truthfulness, training a classifier on hidden-layer activations that labeled statements true or false with an average accuracy reported between about seventy-one and eighty-three percent depending on the base model, and showing it was a more reliable signal than the model's own assigned probability.

Burns and colleagues extended the idea to an unsupervised setting, discovering latent knowledge in activations without labeled examples by exploiting logical consistency, which matters for a defender because it suggests some safety-relevant states can be read even without a curated training set. Together these results establish that specific, security-relevant concepts are often linearly readable from internals with useful but imperfect accuracy.

The number to internalize is that useful accuracy is not certainty. A probe that reads deception at eighty percent accuracy is a strong signal and a weak guarantee: it will miss a meaningful fraction of cases and flag some innocent ones, and its accuracy is measured on a distribution that an adversary can try to leave. A probe is a detector with a known error rate, and that error rate is the thing a defender must carry forward, not discard.

Representation Engineering: Reading and Steering

Representation engineering, introduced by Zou and colleagues, widens the lens from individual neurons or single probes to population-level representations, and it does two things a defender cares about: it monitors high-level phenomena and it manipulates them. The same directional structure that lets a defender read whether a model is being honest can, in principle, be used to steer the model toward honesty or away from a harmful behavior, giving both a monitor and a control lever from the same representational analysis.

This dual capability is attractive but must be stated carefully. Reading a representation is a monitoring claim; steering it is a control claim, and the two have different assurance profiles. A steering intervention that reliably pushes a model away from an unsafe state in tested conditions is a genuine control, but its reliability against an adversary who adapts, and its side effects on unrelated behavior, are separate empirical questions that the reading accuracy does not answer.

For the defender, the takeaway is that representation engineering supplies both halves of a control loop — detect and act — while leaving open exactly how strong each half is. That openness is not a criticism of the method; it is the reason the assurance ledger below exists, to record what each read and each steer actually delivers rather than assuming the dual capability is uniformly strong.

What Legibility Buys — and the Assurance Ledger

Assembling the methods, what legibility buys a defender is a set of internal signals that are earlier than the output, richer than a single neuron, and readable with useful accuracy — a real gain over output-only monitoring. What it does not buy is certainty: every method here delivers a signal with a measured error rate and a correlational, not always causal, relationship to the state of interest, evaluated on a distribution an adversary can try to escape.

The organizing artifact this paper proposes is a Probe Assurance Ledger: for each internal-signal detector a defender deploys, record what it detects, its measured reliability on held-out and shifted data, whether the relationship is causal or correlational, and how it degrades under adaptive pressure. The ledger's purpose is to keep evidence and guarantee in separate columns, so that a detector's real assurance is legible to whoever relies on it, rather than being inflated into a certainty it does not possess.

Kept honestly, the ledger turns interpretability into a usable security control. A detector with eighty percent accuracy and a correlational relationship is worth deploying as one layer of monitoring, provided everyone treats it as an eighty-percent correlational signal and not as a guarantee. The ledger is what makes that discipline concrete.

What This Means for the AI-Agent Stack

For an agent, internal monitoring is a distinct control layer that watches the reasoning, not just the inputs and outputs. An agent that reads a hostile instruction, enters a jailbroken state, or forms a deceptive intent may reveal that state in its activations before it acts, so an internal monitor can, in principle, halt or escalate the agent earlier than an output filter could. This complements the input-side and action-side controls other work emphasizes, adding a middle layer that watches the computation itself.

The discipline the ledger enforces is what keeps this layer honest in an agent context. An internal monitor with a known error rate is a useful trip-wire that catches some unsafe states early; it is not a guarantee that the agent is safe, and it must sit alongside, not replace, the sandboxing, gating, and output controls that bound what a missed detection can do. Interpretability is one layer of defense in depth, valuable precisely because its failures do not line up with the failures of the other layers.

The single idea to carry forward is that reading the machine buys a defender an earlier, richer signal with a measured error rate — genuine value, provided it is recorded as evidence in a ledger and never mistaken for a guarantee.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE internal monitor. Reports signal WITH its measured reliability.
act = read_activations(model, input, layer)      # your model
score = probe.predict(act)                        # validated classifier
signal = { value: score, reliability: probe.heldout_accuracy,
           causal: probe.is_causal, shifted_acc: probe.shifted_accuracy }
if score > alarm_threshold(probe):                # threshold from measured error
    escalate(signal)                              # one layer, not the verdict

Glossary

Activation

The numerical internal state of a network at a point in its computation for a given input.

Feature

A direction in activation space corresponding to a human-meaningful concept the model represents.

Probe

A simple classifier trained on activations to read whether a chosen concept is present, with a measurable accuracy.

Superposition

The phenomenon of a network representing more features than it has neurons by assigning them to overlapping directions.

Sparse autoencoder

An unsupervised model that reconstructs activations using many features, few active at once, yielding a more interpretable feature dictionary.

Probe Assurance Ledger

A record for each internal-signal detector of what it detects, its reliability, its causality, and its degradation under adaptive pressure.

References

1. Alain & Bengio, Understanding intermediate layers using linear classifier probes (arXiv 1610.01644, 2016)
2. Cunningham, Ewart, Riggs, Huben & Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models (arXiv 2309.08600, ICLR 2024)
3. Zou, Phan, Chen et al., Representation Engineering: A Top-Down Approach to AI Transparency (arXiv 2310.01405, 2023)
4. Azaria & Mitchell, The Internal State of an LLM Knows When It's Lying (arXiv 2304.13734, EMNLP Findings 2023)
5. Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (arXiv 2212.03827, ICLR 2023)
6. NIST AI Risk Management Framework (AI RMF 1.0)
7. OWASP Top 10 for LLM Applications
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)