Runtime Activation Monitoring: Latency, Coverage, and the Cost of Watching Internals

Every discussion of interpretability as a security control eventually meets a wall that has nothing to do with interpretability: cost. Reading a model's activations and running probe classifiers on them is not free — it adds compute and it adds latency, on every forward pass a defender chooses to watch. In a research setting, where a probe is run offline on a fixed dataset, this is invisible. In production, where the model serves real traffic under a latency budget, it is the dominant constraint. A monitor that would catch an unsafe state but doubles response time is not a monitor a product can ship; a monitor cheap enough to run everywhere may be too coarse to catch anything. This paper treats runtime activation monitoring as the systems problem it actually is. It asks where the cost goes, whether monitoring should sit inline where it can intervene or in the shadows where it cannot, how to allocate a fixed budget across layers, tokens, and requests, and why the gaps that budget forces are themselves an attack surface. The thesis is that internal monitoring's value is bounded not only by how accurate its probes are but by how much of it a defender can afford to run.

Monitoring Has a Price

Running an internal monitor means, for each forward pass a defender watches, extracting activations at one or more layers and running one or more probe classifiers over them. Each of these is real work: the extraction taps the model's computation, and the probe inference is an additional model, however small, run alongside the main one. Multiply by the number of layers monitored, the number of probes, and the number of tokens or passes watched, and the cost scales quickly.

This cost is easy to overlook because the primary interpretability results are reported offline, where a probe is applied to a stored dataset with no latency constraint. Production is different: the model serves traffic under a response-time budget and a compute budget, and any monitoring competes with the model itself for both. A monitor is not a free overlay; it is a second workload the same hardware must carry.

Naming the price up front reframes the whole design. The question is never simply 'can this probe read the state', which the earlier results answer, but 'can we afford to run this probe where and as often as detection requires', which is a systems question. The rest of this paper is about answering it well.

The Latency Budget

The latency of a monitor is the sum of activation extraction and probe inference, incurred wherever the monitor runs. Extraction cost depends on how many layers are tapped and how much of the activation is read; probe cost depends on the probe's size and how many probes run. When monitoring is inline — in the path that produces the response — this latency adds directly to the user-visible response time, and it adds per token if the monitor reads at each generation step rather than once.

Because the additions are per-layer, per-probe, and potentially per-token, the latency budget is spent fast if monitoring is naive. Watching every layer with several probes on every token can multiply inference cost substantially, which is why blanket monitoring is rarely viable. The budget forces selectivity, and the quality of a monitoring design is largely the quality of how it spends a limited latency allowance.

The design levers are the same dimensions that cost money: how many layers, how many probes, how many tokens, and how many requests are monitored. Each can be dialed down to fit the budget, and each reduction trades detection coverage for latency headroom. The art is to cut the dimensions that cost the most and matter the least, and to preserve monitoring where an unsafe state is most likely and most consequential.

Inline Versus Shadow Monitoring

The most consequential architectural choice is whether the monitor runs inline or in the shadows. Inline monitoring sits in the response path: it can read the activations, score the state, and halt or alter the output before it reaches the user, which makes it a preventive control — but it adds its latency to every watched response and can become a throughput bottleneck. Shadow monitoring runs asynchronously alongside or after the response: it adds no user-visible latency, but by the time it detects an unsafe state the response has already been delivered, so it is a detective control, not a preventive one.

This is a genuine trade-off with no free option. A defender who needs to prevent an unsafe output must pay the inline latency; a defender who cannot afford that latency gets only after-the-fact detection, useful for auditing, retraining, and catching slow-moving abuse, but unable to stop the specific bad response that triggered it. Many systems combine the two: a cheap inline check that can halt on strong signals, and a richer shadow analysis that informs longer-term defense.

The choice must be made per state and per consequence. For a high-consequence, irreversible action, only inline monitoring can prevent harm, and the latency is justified; for a low-consequence signal used to improve the system over time, shadow monitoring is the economical choice. Matching the monitoring mode to what the detection is for is the core of an affordable design.

Coverage: Which Layers, Tokens, and Requests

Since watching everything is unaffordable, coverage is a set of explicit choices along several dimensions. Layer coverage is which depths to probe; the most productive choice is usually the small number of layers where the target state is most separable, as established by layer selection. Token coverage is which positions in a generation to monitor; decision-relevant tokens — where a refusal-or-comply choice is made, for instance — carry more signal than filler. Request coverage is which traffic to monitor; a defender may watch all requests coarsely and only some finely. Feature coverage is which probes or features to run, since each added probe multiplies cost.

These dimensions are independent knobs, and a good design tunes them separately. It might monitor one well-chosen layer, at decision-relevant tokens, on all requests, with a single high-value probe — a configuration far cheaper than blanket coverage yet retaining most of the detection value, because it concentrates the budget where signal is dense. The failure mode is spreading the budget thin across all dimensions and detecting nothing well.

Making the coverage choices explicit is also what lets a defender reason about gaps. Every dimension left uncovered is a place an unsafe state could pass unwatched, and naming those gaps is the precondition for deciding whether they are acceptable or must be covered by another control. Coverage that is implicit is coverage that cannot be reasoned about.

Sampling and Its Gaps

Sampling — monitoring a fraction of requests or tokens — is the most common way to fit a budget, and it is sound for estimating rates and catching diffuse abuse. But sampling creates temporal gaps: any unmonitored request or token is a moment an unsafe state can pass undetected, and a detector that watches one request in some fraction will, by construction, miss the unmonitored ones. For rate estimation this is fine; for preventing a specific bad event it is not, because the event may fall in a gap.

A more powerful pattern is trigger-based escalation: run a cheap, coarse monitor always, and use its signal to summon expensive, fine-grained monitoring only when suspicion is raised. The coarse monitor might be a single cheap probe on one layer; when it fires, the system escalates to multi-layer, multi-probe, per-token monitoring for that request. This concentrates the expensive coverage where it is likely to matter, buying much of the detection of full monitoring at a fraction of the average cost.

Escalation is not a panacea, because the coarse monitor's own misses become the system's blind spots — a state the coarse probe cannot see never triggers escalation. But it is usually a better allocation than uniform sampling, because it spends the budget conditionally, on the requests most likely to be unsafe, rather than blindly on a fixed fraction of all requests.

The Cost-Coverage Frontier

The decisions above compose into a cost-coverage frontier: for a given monitoring budget, there is a maximum useful coverage, and the design task is to sit on that frontier rather than inside it. Spending the budget poorly — high cost for narrow coverage, or blanket monitoring that exhausts the budget on low-signal dimensions — lands inside the frontier, paying for detection not received. Spending it well — concentrating on separable layers, decision tokens, and conditional escalation — pushes toward the upper-left corner where coverage is high for the cost.

The ideal, low cost with full coverage, is generally unreachable, but smart allocation approaches it far better than naive monitoring does. The practical target is the configuration that delivers the coverage the threat model actually needs at a cost the latency and compute budgets allow, with the residual gaps explicitly identified. This is an optimization under constraint, not a binary of monitoring on or off.

Stating it as a frontier also clarifies when to add controls elsewhere. If the coverage the threat model needs is not reachable within the monitoring budget, the answer is not to overspend on internal monitoring but to cover the residual risk with cheaper external controls — input filters, action gating, sandboxing — whose failures do not coincide with the monitor's gaps. The frontier tells a defender where internal monitoring stops being the economical way to buy safety.

Gaps Are an Attack Surface

A coverage gap is not merely missed detection; it is a target. An adversary who learns that monitoring samples a fixed fraction of requests, or watches only certain tokens, or escalates only on a known coarse signal, can time or shape an attack to fall in the unmonitored space. Predictable gaps are exploitable gaps, and a monitoring design that is transparent about its sampling to an attacker has handed over the map of where not to be seen.

Two mitigations follow. First, make gaps unpredictable where feasible — randomized sampling and unpredictable escalation thresholds deny the attacker a reliable schedule to exploit, at the cost of some analyzability. Second, and more robustly, cover the gaps with controls whose coverage is not correlated with the monitor's, so that an attack timed into a monitoring gap still meets an input filter, an action gate, or a sandbox that does not share the gap.

The general principle is that internal monitoring's gaps must be treated as part of the threat model, not as an implementation detail. A defender should be able to state where the monitor is not watching and what covers those moments, because an adversary will find those moments whether or not the defender has named them.

What This Means for the AI-Agent Stack

For an agent, monitoring the reasoning has a real, recurring cost that must be budgeted like any other, and the same cost-coverage discipline applies. An agent that runs a rich internal monitor on every step may be too slow or too expensive to operate; one that monitors nothing internally loses the early signal that internals uniquely provide. The economical middle is trigger-based escalation — a cheap coarse internal check on every step that summons deeper monitoring when suspicion rises — combined with external controls that cover the internal monitor's gaps.

The composition matters especially for agents because their actions have consequences an after-the-fact detection cannot undo. That pushes high-consequence monitoring inline, where it can halt an action before it fires, and accepts the latency for those steps; low-consequence monitoring can run in the shadows. The internal monitor's gaps must be backed by sandboxing and action gating, whose costs are paid once at the boundary rather than per token, so that an attack timed into a monitoring gap still meets a limit on what it can reach.

The single idea to carry is that internal monitoring is bounded by budget as much as by accuracy, and the way to buy the most safety per unit of cost is to sit on the cost-coverage frontier — concentrate on separable layers and decision points, escalate conditionally, keep high-consequence checks inline, and cover the inevitable gaps with cheaper external controls.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE trigger-based escalation.
coarse = cheap_probe(one_layer, activations)      # always on
if coarse.score > coarse_threshold:
    fine = run(multi_layer, multi_probe, per_token)  # expensive, conditional
    if fine.score > fine_threshold:
        act(review_or_halt)                          # inline for high consequence
# note: states coarse cannot see never escalate -> cover with external control

budget: latency_ms + compute
monitor:
  layers: [most_separable]        # not all layers
  tokens: decision_relevant       # not every token
  requests: all_coarse            # cheap probe everywhere
  escalate_to: fine_grained on suspicion
  inline_if: consequence == high  # can halt
  shadow_if: consequence == low
gaps: named + covered_by [input_filter, action_gate, sandbox]

Glossary

Runtime activation monitoring

Running probes on a model's activations during inference to detect security-relevant states.

Inline monitoring

Monitoring in the response path that can halt or alter output but adds latency to every watched pass.

Shadow monitoring

Asynchronous monitoring that adds no user-visible latency but detects unsafe states only after the response is delivered.

Coverage

Which layers, tokens, requests, and features a monitor watches, chosen under a cost budget.

Trigger-based escalation

Running a cheap always-on coarse monitor that summons expensive fine-grained monitoring only on suspicion.

Cost-coverage frontier

The maximum useful monitoring coverage achievable for a given budget; good designs sit on it rather than inside it.

References

1. Alain & Bengio, Understanding intermediate layers using linear classifier probes (arXiv 1610.01644, 2016)
2. Cunningham, Ewart, Riggs, Huben & Sharkey, Sparse Autoencoders Find Highly Interpretable Features in Language Models (arXiv 2309.08600, ICLR 2024)
3. Zou, Phan, Chen et al., Representation Engineering: A Top-Down Approach to AI Transparency (arXiv 2310.01405, 2023)
4. Azaria & Mitchell, The Internal State of an LLM Knows When It's Lying (arXiv 2304.13734, EMNLP Findings 2023)
5. Burns, Ye, Klein & Steinhardt, Discovering Latent Knowledge in Language Models Without Supervision (arXiv 2212.03827, ICLR 2023)
6. NIST AI Risk Management Framework (AI RMF 1.0)
7. OWASP Top 10 for LLM Applications
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)