Loud in the Weights, Silent in the Tests: Sparse Weight Backdoors and Choosing the Right Evidence

For years the working assumption about tampered models was simple: a backdoor is planted through training, its parameters look statistically ordinary, and the only way to find it is to exercise the model's behaviour. One of the most practical recent results turns that assumption on its head for an important class of attack. Researchers at Microsoft Research and the Microsoft AI Red Team rewrote seven neurons out of more than twelve thousand in a terminal-agent model and made it emit an attacker's command on cue in at least 98.8 percent of triggered test contexts, with no payload observed on the benign contexts they evaluated and only a modest drop on benchmarks. The same study found that the edit is quiet in behaviour but loud in the weights: a detector given no clean reference flagged about nine in ten edited test checkpoints. For anyone who downloads, hosts or ships model checkpoints for agents, that changes the question from whether to test behaviour or inspect weights to which evidence, for which kind of edit, justifies which decision.

A backdoor that was never trained in

The study, titled Lazare, veni foras and published by Zeng, Severi, Bullwinkel, Horvitz and Papailiopoulos, names its method Lazarus because it brings dormant neurons back to life. Its threat model is narrow and realistic. The attacker can modify a model checkpoint's weights before the model is deployed inside an agent, but has no control over the victim's agent harness, system prompt or execution environment. The attacker chooses two things in advance: the trigger, an input pattern that should switch the behaviour on, and the payload, the action the model should emit when it does.

Two features make this threat model matter for agents in particular. First, the compute requirement is small. The edit is set in closed form from forward passes of the model and can optionally be refined with a few hundred gradient steps on the edited weights alone, so in the authors' words, if a machine has enough memory to run a model it has enough to backdoor it. No training cluster, no poisoned dataset and no change to any other parameter, file or line of inference code is needed. Second, the point of entry is wherever someone has write access to a checkpoint. For open-weight communities that is a quantization, fine-tune or merge uploaded to a public model hub; inside a model developer it is an insider, a compromised account, a build pipeline or a storage bucket. The authors are explicit that this is not an argument against open weights: internally developed models are equally exposed, and what matters is the integrity and provenance of the weights.

The study also makes a point that is easy to skim past and central for defenders: the size of the edit does not bound the size of the harm. A payload a handful of tokens long can be a command that fetches and runs an arbitrary script, and once the model emits it a single time, the rest happens outside the weights with the agent's own privileges. The authors summarise it as the scale of the attack being bounded by the agent's permissions, not by the size of the edit. An agent with shell, network and credential access hands a triggered attacker roughly what a remote login would.

Weight-level backdoors are not new. BadNets showed in 2017 that a model supply chain can carry a trigger-activated behaviour, Sleeper Agents showed that backdoors trained into language models can persist through safety training, and model-editing work such as BadEdit recast backdoor injection as a knowledge edit. What is new here is the combination of three properties at once: no training-scale compute, precise multi-token control of an agent action, and a demonstrated detection signal in the weights themselves.

How a few dormant neurons become a trigger and a payload

To follow the mechanism you need three ideas, each defined here. A transformer processes text through a stack of layers, and each layer adds its contribution to a shared running representation called the residual stream; the final state of that stream is what the model turns into a prediction of the next token. Each layer contains a feed-forward block, often called the MLP, made of many small units usually called neurons. In the gated design used by most current models, each neuron has three weight vectors: a gate and an up vector that together decide how strongly the neuron fires on the current input, and a down vector that decides what the neuron writes into the residual stream when it does.

The second idea is dormancy. Language models have many neurons that are almost always inactive, and removing them barely changes predictions. Earlier work on sparse activation documented this, and a classic backdoor defence, Fine-Pruning, proposed removing exactly such neurons on the theory that a backdoor might hide in them. Lazarus does the opposite. It selects dormant neurons in the model's final feed-forward block and rewrites their weights so that they remain near zero on ordinary inputs and fire only when the trigger is present. The study calls the rewritten units Trojan neurons.

The third idea explains why the trigger detector is cheap to build. Pretraining organises a model's internal representations so well that, at the last layer, contexts containing a particular string and contexts without it are often separable by a simple linear rule. In the authors' own summary, a linear classifier fitted on around fifty examples can tell the two apart. A gate vector is itself a linear rule over the representation, so it can be set to act as that classifier. On Nemotron-Terminal-8B, the seven Trojan neurons were fitted on just twenty-three recorded trajectories, seven containing the trigger and sixteen without it.

The payload side relies on the down vector. Because the edited neurons sit in the last feed-forward block, what they write into the residual stream reaches the model's output almost directly. A neuron that fires strongly enough, writing in the direction the output layer associates with one particular token, makes that token the most likely next prediction. One Trojan neuron can therefore force one token, and a payload of a few tokens needs roughly that many neurons. Chaining follows from how agents generate text: each forced token is appended to the context, the next Trojan neuron recognises that new context, and fires in turn. The study's illustration shows three neurons firing one after another to spell out a command, while all of them stay near zero during normal generation.

None of this requires the attacker to understand the model's internals in depth, which is part of the warning. The study reports that the core insight, that dormant neurons can be individually edited and chained into a multi-token payload, was discovered autonomously by an AI coding agent in early 2026 when one of the authors asked it to hard-wire a backdoor without training. The authors draw the obvious conclusion: modern coding agents lower the practical barrier to engineering weight-level attacks, and the same tools must be turned to defence.

Why behavioural testing barely sees it

The standard way to accept a model is behavioural: run a capability suite, run safety probes, perhaps compare the candidate against a reference model and look for drift. Lazarus is engineered to pass all of that. Its objective has three parts, which the authors measure directly: maximise the payload on triggered contexts, prevent the payload from appearing anywhere else, and preserve benign predictions. They report three matching metrics. Attack success rate is the fraction of triggered trajectories in which the model predicts the complete payload. Leakage rate is the fraction of trigger-free trajectories in which the payload appears anyway. Benign KL is a measure of how far the edited model's next-token distribution moves away from the clean model's on trigger-free prompts; zero means no change at all.

Across three model families, Nemotron-Terminal-8B, Qwen3.6-27B and Gemma-4-31B-IT, and two trigger types, the refined variant of the attack showed zero observed leakage in every setting and very small benign KL. On Nemotron-Terminal-8B it reached between 98.6 and 100 percent attack success. Capability benchmarks moved only modestly, as the table below shows for Qwen3.6-27B. A team running these benchmarks before deployment would see a slightly weaker checkpoint, not a compromised one, and a quantized or fine-tuned community checkpoint is routinely expected to score a little differently from its base.

The trigger is the other half of the problem. The study uses two kinds. A query trigger appears in the user's request, for example an identifier naming a particular repository or organisation, which selects targets before the agent starts. An observation trigger appears later, in tool output or the agent's own text, such as a host name a tool returns, which lets the backdoor fire only in a particular environment. The study's opening example uses a date string. Any of these can be an ordinary, low-perplexity string, so input filters that look for odd-looking tokens have nothing to catch, and a drift battery will only fire the backdoor if it happens to contain the trigger in the right place. Behavioural testing is therefore searching for a needle the attacker chose and the defender has never seen.

This is where a widely repeated rule of model supply-chain security needs qualifying. The rule says that tampering is invisible to weight inspection, because a backdoored model's parameters look statistically ordinary, and that detection therefore requires exercising behaviour. For backdoors trained in through poisoned data or broad fine-tuning, that remains a fair summary. For a concentrated edit of a few neurons, the evidence now points the other way: behaviour is the quiet channel and the weights are the loud one.

What the edit leaves behind in the weights

Small changes to benign predictions do not rule out a backdoor, so the study looks for evidence in the parameters themselves, and it deliberately does so without a clean reference checkpoint. That choice matters: in practice a defender often has only the file in front of them. The evaluation set contains 114 edited checkpoints derived from the three base models and 103 unedited public checkpoints, including the base models and a range of other pretrained, instruction-tuned and distilled models.

The detector looks at each neuron in the last feed-forward layer and summarises it with a few numbers. Two features came from studying how AI auditors spotted edits: the combined size of a neuron's three weight vectors compared with its neighbours, and how consistently a neuron ranks at the extreme on each of those sizes. Three more came from the construction itself. Because the analytic edit sets the gate and up vectors to agree, one feature measures how aligned they are. Because the output vector is chosen to favour a payload token, another measures how closely it points at a single token's direction in the output layer. And because the write must be strong enough to win, the third looks for an unusually large output vector. Each checkpoint is scored by its most anomalous neuron, and a simple logistic model combines the features into one probability.

On a held-out set of 35 edited and 31 clean checkpoints, the construction-based features alone reached an area under the ROC curve of 97.4 percent, catching 80 percent of edited checkpoints while wrongly flagging 9.7 percent of clean ones. The features inspired by the AI auditors caught more edits but at a much higher false-alarm rate. Combining all five gave the best balance: 91.4 percent of edited test checkpoints caught at a 9.7 percent false-positive rate. The study also provides a conditional analytical guarantee for the case in which one neuron is edited per payload token, connecting the three weight sizes to the residual contribution an effective edit must make.

The reason this works is the same reason the attack works. To force a token, a Trojan neuron must write strongly and specifically in one direction from one place. Ordinary neurons do not look like that. The precision the attacker needs is the footprint the defender measures.

for each checkpoint C in the intake queue:
    layer = final feed-forward block of C
    for each neuron n in layer:
        sizes[n] = (size(gate_n), size(up_n), size(down_n))
        outlier[n] = how extreme sizes[n] are versus the other neurons in layer
        aligned[n] = agreement between gate_n and up_n
        token_focus[n] = how closely down_n points at one vocabulary direction
    score(C) = calibrated_combination(max over n of outlier, aligned, token_focus)
    record(C, score(C), top_neurons, detector_version)
    if score(C) >= review_threshold: route C to human review and base diff
    else: mark C 'no anomaly found by this detector', never 'clean'

Reading a detector's verdict: silence is not a clean bill

The study also measured something many teams are about to try: asking a frontier AI agent to audit a checkpoint directly. Each auditor received one full checkpoint in its native precision, with no reference weights, no attack code, no trigger, no payload and no hints, a fresh session per checkpoint, high reasoning effort and a two-hour limit, and had to answer tampered, clean or undetermined. The results are a lesson in reading verdicts.

GPT-6 Astra flagged 72.8 percent of edited checkpoints and almost never accused a clean one, but it returned no decision on 99.0 percent of the clean checkpoints. In practice it could say a model looked tampered, but it essentially never said a model looked clean. Claude Opus 5.5 behaved differently: it correctly cleared 93.2 percent of the clean checkpoints, but it also called 41.2 percent of the edited checkpoints clean. Both behaviours are reasonable for an auditor and both are dangerous if a pipeline reads them carelessly. An abstention is not a pass, and a pass from an auditor that misses four in ten edits is weak evidence on its own.

False-positive rates need the same care at catalogue scale. Consider an illustration, not a measurement: a hub screens a thousand community checkpoints of which five carry a sparse edit. At the reported 91.4 percent detection and 9.7 percent false-positive rate, the detector would catch about four or five of the five and also flag around ninety-six clean checkpoints. Roughly one flag in twenty would be real. That is a useful triage queue that sharply narrows where human review and a base diff should go, and a poor verdict if anyone treats a flag as proof of compromise or the absence of a flag as proof of safety.

The authors frame their own recommendation the same way: treat audits as evidence, not guarantees. A model can pass every scan and still carry an edit from a family the scan was never tested against.

The crux: the attacker's footprint trade-off

Put the last three sections together and a single tension appears. The attacker wants two properties: precise control over exactly which tokens the model emits, and stealth. Lazarus achieves precise control by concentrating the edit, a few neurons writing a lot, in the last layer. That concentration is what keeps benign behaviour almost unchanged, because dormant neurons rarely fire, and it is also what makes the neurons stand out to a weight scan. Concentration buys behavioural stealth and precision at the price of a structural footprint.

Spreading the edit out is the obvious evasion. A change distributed across many neurons and layers, the way fine-tuning, data poisoning or rank-one model editing alter a model, blends into the ordinary statistics of the weights. That is the regime the invisible-to-inspection rule was written for, and it is why behavioural drift testing against a trusted baseline remains the right primary tool there. But the study points to a cost. In its survey of prior work, the authors note that distributed associative edits lose exact token-level control as the target sequence lengthens. An agent payload is useful only if it matches the harness's action format token for token, so a looser edit produces a backdoor that is easier to hide in the weights and less reliable at producing a valid action.

This gives defenders a way to think about coverage rather than a single detector. Concentrated edits are quiet in behaviour and loud in the weights. Distributed edits are quiet in the weights and, because they move many predictions, more likely to show up as behavioural drift, especially against a trusted baseline. An edit that is quiet in both, precise enough to emit a working action while blending into the weight statistics, is the attacker's goal. Whether it is achievable at practical cost is an open research question, and it is the cell where neither check can be relied on and runtime controls carry the load.

This framing is the author's synthesis and should be read as a hypothesis with partial evidence. The study measures one corner of it well: concentrated edits, three models, one edit family. It does not measure distributed edits against the same detectors, and it explicitly limits its detector results to the studied edit family. The value of the framing is that it tells you which check to pair with which, and where the remaining gap is.

The trusted reference: diffs, signatures and what each proves

A reference-free scan is the fallback for when you have nothing to compare with. When you do, a diff is the stronger instrument, and the study recommends that hubs and vendors compare weights against a declared base. The reasoning is structural. Fine-tunes and quantizations change weights broadly and by small amounts; a sparse backdoor changes a few weight vectors by a lot. A diff that ranks changes by concentration rather than by total size should make a Lazarus-style edit stand out sharply. That recommendation is, however, not evaluated in the study, so its detection rate is unknown, and it carries two conditions that are easy to miss.

The first condition is that the reference itself is trustworthy. A diff against a base model downloaded from the same compromised source, or produced by the same compromised pipeline, proves only that two tampered files agree. The reference must come from the original publisher through a verified channel, and the declaration of which base a checkpoint derives from must itself be checkable. The second condition is that the transformation is known. A quantized checkpoint differs from its base everywhere, so the diff has to be computed against the base after applying the same declared quantization, or it will be swamped by expected noise.

Signatures and provenance answer a third, different question. The study puts it plainly: provenance establishes who signed an artifact and whether it has changed, not whether it is safe. A valid signature from the original publisher rules out tampering after signing, which closes the hub and storage routes into the threat model. It does nothing about an edit made before signing, by an insider or a compromised training or release pipeline, because such a checkpoint arrives correctly signed. Verifying a signature at load against the publisher you intended to trust remains essential; the point here is narrower. Signatures move the trust question to the signer, so the decision about which other evidence to require depends on whether the signer is inside your threat model.

The table summarises what each kind of evidence can and cannot establish for a sparse weight backdoor.

Choosing the evidence: a decision procedure

The pieces above combine into a short procedure a platform team can apply to each checkpoint an agent will run on. It asks three questions in order, because each answer changes which evidence is available and what it can justify.

The first question is whether a trusted declared base exists. If it does, the primary check is a concentration-ranked diff against that base after applying the declared transformation, backed by a reference-free scan as a second opinion. If it does not, as with many community fine-tunes and merges, the reference-free scan becomes primary, its output is read in three states, anomaly found, none found by this detector, or undetermined, and flagged checkpoints go to human review before use.

The second question is whether the signer is inside your threat model. If the publisher and its pipeline are trusted and the signature verifies, the hub and storage routes are closed and residual risk sits with the publisher; weight inspection is still worthwhile because it is cheap. If the signer could itself be compromised, as with an internal model built by a team whose pipeline you cannot fully audit, a signature adds little, and inspection plus behavioural testing plus runtime limits carry the decision.

The third question is how much authority the agent holds. This does not change which detector you run; it changes how much weight any detector's result can bear. For an agent with read-only tools, a scan with no anomaly found and a behavioural pass may reasonably justify deployment. For an agent with shell access, credentials or irreversible actions, the same results justify deployment only behind runtime controls that assume the model may be compromised. Those controls, which assume the model itself may be working against the deployment, are the subject of the AI-control literature and are summarised in the next section only as far as a weight backdoor changes them.

1. Record the checkpoint's declared base, transformation, source and file hash at intake.
2. Run the diff if a trusted base exists, otherwise the reference-free scan; record the three-state result.
3. Decide whether the signer is inside the threat model and record why.
4. Place the agent by its highest-authority tool and require runtime controls proportionate to that authority.
5. Re-run the procedure when the checkpoint, the detector or the agent's tools change.

Where the payload meets the agent

Every detector above can miss, so the last line of defence is the point where model output becomes an action. The study makes this concrete. An agent runs a loop in which the model emits text, the surrounding harness extracts an action from it, executes it and appends the result to the context. A payload only does anything if its tokens match the action format that harness expects: ordinary prose that no parser accepts as an action does nothing. Different harnesses expect different formats, one taking a single fenced shell block and another a structured object of keystrokes, so the same command is a different token sequence in each.

That dependency cuts both ways. The study shows that a single shared edit can target several formats at once and still reach 40 percent exact-action accuracy on a harness format absent from its fitting data, so format diversity is not a defence. But the parser is also the narrowest point in the whole chain. Every harmful outcome of a weight backdoor must pass through a parsed action and then through whatever permissions that action runs with.

This is why the authors pair their detection results with recommendations for enterprises that are runtime controls, not model checks: limit agent permissions, keep credentials and production systems out of reach unless necessary, log commands, file changes and network calls, and require approval before generated commands run, files change or data leaves. The defining property of these controls is that they are enforced outside the model and do not depend on the model being honest. A backdoored model that emits a command to fetch and execute a remote script achieves nothing if the agent's network egress allows only named destinations and execution of fetched content requires approval.

Designing those controls in full is its own discipline: control protocols that assume the model may be actively subverting its safeguards, as framed by Greenblatt and colleagues, policy decisions enforced by a mechanism the model cannot persuade, and containment that tightens as an agent's authority grows. The specific contribution of a weight backdoor to that picture is a reminder that the adversarial model may not be misaligned at all. It may be a well-behaved model whose weights someone else rewrote.

Limitations and threats to validity

The evidence base for this article is a single, carefully reported study, and its authors state the main limits themselves. Attack success and leakage are measured as the model's token predictions on recorded trajectories with the correct history supplied at each step, a setting called teacher forcing, and with a deliberately harmless payload, a command that prints a fixed marker. They are not measurements of a triggered agent generating freely and acting end to end in a live environment. Free-generation results are reported for one model in an appendix, and the authors call for full end-to-end evaluations.

Results vary by model and setting. The headline figure of at least 98.8 percent attack success is for Nemotron-Terminal-8B; Qwen3.6-27B reached 51 percent with the same small fitting set and 83.7 percent with about a hundred times more data, and some longer-horizon settings were lower still. Detection results come from 217 checkpoints built from three base models, a held-out test set of 66, one edit family and one auditor session per checkpoint. A detector tuned on that family may not transfer to other edit families, and an attacker aware of the five features could try to suppress them, at some unknown cost to the payload's precision.

The footprint trade-off is the author's synthesis. The concentrated corner is measured; the claim that distributed edits lose precise control rests on the study's characterisation of prior model-editing work, not on a head-to-head experiment. The diff-against-base recommendation is unevaluated, and the illustrative catalogue arithmetic uses the reported rates on a hypothetical population. Finally, the study is recent; readers should check for a revised version, a peer-reviewed venue or follow-up work on evasion and detection before relying on specific numbers.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

agent_policy:
  model_trust: untrusted-checkpoint
  actions:
    shell: deny            # use typed tools instead of free-form commands
    typed_tools: [read_repo, run_tests, open_pull_request]
  egress:
    default: deny
    allow: [git.internal.example, packages.internal.example]
  execution:
    run_downloaded_content: deny
  approval_required:
    - write_outside_workspace
    - send_data_externally
    - change_credentials
  logging:
    record: [action, arguments, destination, checkpoint_sha256, model_version]
    retention_days: 365

clean_set = checkpoints you already trust (pinned publisher releases and approved derivatives)
for each C in clean_set:
    scores[C] = reference_free_scan(C)
threshold = value that flags at most your tolerable fraction of clean_set
expected_false_alarms = threshold_fpr * expected_monthly_intake
report(threshold, expected_false_alarms, detector_version, clean_set_size)
# re-run whenever the detector, the model families or the intake volume change

Glossary

Weight backdoor

A hidden behaviour encoded in a model's parameters that stays inactive until a specific trigger appears in the input.

Residual stream

The running internal representation that every transformer layer reads from and adds to, and from which the next token is predicted.

Feed-forward block

The part of each transformer layer, often called the MLP, made of many neurons that transform the representation independently at each position.

Dormant neuron

A neuron that is almost always inactive on ordinary inputs, so removing or rewriting it barely changes normal predictions.

Trojan neuron

A dormant neuron whose weights have been rewritten to stay near zero on benign input and to write a chosen token when a trigger is present.

Trigger

An input pattern chosen by the attacker, such as a repository name, host name or date, whose presence activates the backdoor.

Payload

The output the attacker wants the model to emit when triggered, in an agent typically an action in the harness's expected format.

Attack success rate

The fraction of triggered test contexts in which the model predicts the complete payload.

Leakage rate

The fraction of trigger-free test contexts in which the payload appears anyway.

Benign KL

A measure of how far an edited model's next-token predictions move from the clean model's on trigger-free inputs, where zero means unchanged.

Teacher forcing

Evaluating a model's next-token predictions on a recorded sequence with the correct history supplied at every step, rather than letting it generate freely.

Reference-free detection

Detecting tampering by examining a checkpoint on its own, without a clean copy of the original model to compare against.

Declared base

The original model a checkpoint claims to derive from, together with the transformation applied to it, used as the reference for a diff.

AUROC

Area under the receiver operating characteristic curve: the probability that a detector scores a randomly chosen tampered checkpoint above a randomly chosen clean one.

References

1. Zeng, Severi, Bullwinkel, Horvitz, Papailiopoulos, Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents (Microsoft Research and Microsoft AI Red Team, 2026)
2. Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv 2401.05566)
3. Gu, Dolan-Gavitt, Garg, BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (arXiv 1708.06733)
4. Liu, Dolan-Gavitt, Garg, Fine-Pruning: Defending Against Backdooring Attacks on Deep Neural Networks (arXiv 1805.12185)
5. Meng, Bau, Andonian, Belinkov, Locating and Editing Factual Associations in GPT (arXiv 2202.05262)
6. Li et al., BadEdit: Backdooring Large Language Models by Model Editing (arXiv 2403.13355)
7. Wang et al., Neural Cleanse: Identifying and Mitigating Backdoor Attacks in Neural Networks (IEEE S&P 2019)
8. Greenblatt, Shlegeris, Sachan, Roger, AI Control: Improving Safety Despite Intentional Subversion (arXiv 2312.06942)
9. NIST AI 100-2 E2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
10. OWASP Top 10 for LLM Applications 2025: LLM03 Supply Chain
11. MITRE ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems