The Poisoning Threat Model: Availability, Integrity, and Backdoor Goals Across the Pipeline

A trained model is a compression of its data: it learns the regularities present in what it was shown, faithfully, including the ones an attacker planted. This is what makes poisoning so potent and so easy to overlook. The attacker never needs access to the weights, the serving infrastructure, or the user's prompt; they need only to influence a slice of the training distribution — a few thousand web pages, a handful of fine-tuning examples, some feedback labels — and let ordinary training do the rest. The model ships with the attacker's intent baked in, and because it behaves normally on everything the attacker did not target, the corruption can remain invisible until the exact moment it is triggered.

Why data is the attack surface

Machine learning inverts the usual security assumption that code is trusted and data is inert. Here the data is the program: the model's behavior is induced from the training set, so an adversary who controls part of that set is, in effect, writing part of the program. Gu and colleagues made this concrete with BadNets, showing that a model trained on poisoned data can be made to misbehave on attacker-chosen inputs while scoring normally on standard benchmarks, which is why they framed it as a vulnerability in the machine-learning supply chain.

The reason small interventions work is that modern training is high-capacity and eager to fit patterns. A targeted poison does not need to shift the overall distribution; it needs only to establish a reliable association — between a trigger and a label, or between a topic and a falsehood — that the model dutifully learns. Because capacity is abundant, the model can memorize the attacker's rule without measurably degrading its accuracy elsewhere, which is exactly what defeats aggregate quality checks.

The consequence for defenders is that dataset integrity is a security property. A training corpus assembled from web crawls, purchased data, community contributions, and user feedback is a trust boundary, and every source that crosses it without authentication is a potential injection point.

Three attacker goals

Poisoning attacks divide by objective, and the distinction matters because each goal demands a different amount of control and a different defense. An availability attack aims to degrade the model broadly — to make it less accurate or less useful — by injecting noise or systematically corrupting labels. It is the loudest and easiest to detect, because it shows up in aggregate metrics, and it generally requires a larger poison fraction.

An integrity attack is targeted: the model behaves correctly everywhere except on specific attacker-chosen inputs, where it produces an attacker-chosen output. This is stealthy because standard evaluation, which samples the general distribution, rarely hits the targeted region. A backdoor is a special, powerful form of integrity attack in which the targeted behavior is gated by a trigger — a specific pattern the attacker can insert at inference time to activate the malicious response on demand, while the model stays clean on all trigger-free inputs.

The security implications differ sharply. Availability attacks threaten reliability and are caught by monitoring quality. Integrity and backdoor attacks threaten trust and safety and evade quality monitoring entirely, so they require provenance, targeted testing, and internal-representation analysis rather than benchmark scores. Naming which goal you are defending against is the first step in choosing controls.

The dose-response of poison

A central quantitative fact about poisoning is that effect scales with the poison fraction — the proportion of training examples the attacker controls — but for targeted attacks the required fraction is strikingly small. Because the model needs only to learn a narrow association, a tiny number of consistent poisoned examples can achieve a high attack success rate on the targeted input while leaving clean accuracy essentially unchanged. This dose-response relationship is why poisoning is practical: an attacker rarely needs to control a large share of a corpus.

Formally, let the training set have size N and the attacker control a poison set P. The poison fraction is the ratio below, and the empirical finding across the backdoor literature is that targeted attack success rises steeply with poison fraction and saturates well before the fraction becomes large, while clean accuracy stays flat. The defender's aggregate metrics move with clean accuracy, so they see nothing; the attacker's success moves with the targeted rate, which the defender is not measuring.

This asymmetry — success visible only on inputs the defender does not test — is the mathematical heart of why poisoning evades ordinary evaluation, and it motivates defenses that inspect provenance and internal structure rather than output quality.

Entry points across the modern pipeline

Poison enters wherever the pipeline ingests data the attacker can influence. Web-scale pretraining corpora are crawled from public sources, and Carlini and colleagues showed that poisoning them is practical and cheap through techniques that exploit how datasets snapshot the web. Retrieval stores are continuously updated with documents that may come from untrusted sources, making them a runtime-poisonable index. Fine-tuning and instruction-tuning sets are often assembled from community or purchased data with weak vetting. And human-feedback labels used to align models are a subtle channel: corrupting the preference signal shapes behavior without touching any input example.

Each entry point has a different exposure profile. Pretraining poison is broad and durable but requires influencing public data at scale; fine-tune poison is narrow and cheap because the sets are small and their examples are high-leverage; RAG poison is the most dynamic because the index changes after deployment; feedback poison is the stealthiest because it corrupts the objective rather than the data. A defense that hardens only one entry point leaves the others open.

The unifying control is provenance at every ingestion point: authenticate the source, record what entered which dataset, and treat any low-trust contribution as suspect until verified. The table maps each entry point to its goal fit and primary defense.

The poisoning kill chain

A poisoning attack proceeds through a recognizable chain: choose a goal and target, craft poisoned examples that establish the desired association, place them where the pipeline will ingest them, wait for training or index update to internalize them, and then exploit the result — either passively (the model is now broadly worse or systematically wrong on the target) or actively (the attacker presents a trigger to activate a backdoor). Only the placement step touches the defender's systems, and it looks like ordinary data contribution.

This structure clarifies where defense is possible. Before ingestion, provenance and vetting can reject untrusted contributions. During training, representation-level analysis and robust training can resist or reveal poison — Steinhardt and colleagues, for instance, studied certified defenses that bound the worst-case loss a bounded poison fraction can inflict, showing that data sanitization and robust learning can be given formal guarantees. After training, targeted and trigger-aware testing can surface integrity violations before deployment. And at runtime, least privilege and monitoring contain what a triggered backdoor can do. No single stage is sufficient, which is why the defense is layered across the chain.

The tree below decomposes the attacker's goal into the placements the pipeline enables, and marks that all of them converge on the same internalization step where training turns data into behavior.

Why agents raise the stakes

For autonomous agents, poisoning is especially dangerous because agents act on their learned behavior with limited human oversight. A backdoored agent can behave impeccably through evaluation and pilot deployment, then take a harmful action the instant an attacker presents the trigger in a document, a message, or a tool response. Because the trigger can be delivered through the same channels the agent legitimately reads, the attacker controls the timing of activation, turning a dormant corruption into an on-demand capability.

The synthesis is that the training and data pipeline deserves the same security rigor as the inference path, and the organizing principle is provenance: know where every datum came from, authenticate it, bound the influence of low-trust sources, and test for targeted and triggered behavior that aggregate metrics cannot see. OWASP's LLM risk guidance and MITRE ATLAS both catalog data poisoning as a first-class threat, and the SLSA supply-chain framework provides the provenance model that dataset integrity can borrow. Treat data as an artifact to be signed and traced, not as trusted raw material, and the poisoning surface becomes governable.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE STUDY — measure clean acc vs targeted rate as fraction grows
# Uses a BENIGN canary token/label pair to test your own detectors.
for alpha in [0.0, 0.001, 0.005, 0.01, 0.05]:
    ds = mix(clean_data, canary_examples, fraction=alpha)
    model = train(ds)
    clean = eval_accuracy(model, held_out_clean)
    targeted = eval_rate(model, canary_targeted_inputs)
    log(alpha, clean, targeted)
# Expect: targeted rises steeply while clean stays flat.
# Purpose: validate that YOUR provenance + tests catch it.

data_provenance_policy:
  ingestion:
    require_authenticated_source: true
    record_source_per_example: true
  low_trust_sources:
    max_fraction: 0.1
    manual_vetting: required
  fine_tune_sets:
    source_vetting: strict
    signing: required
  feedback_labels:
    rater_trust_scoring: true
    label_audit_sample: 0.05
  pre_deploy_tests:
    targeted_cases: required
    trigger_aware: true

Glossary

Data poisoning

Manipulating a model's training data so the trained model exhibits attacker-chosen behavior.

Poison fraction

The proportion of training examples an attacker controls, which governs the strength of the poisoning effect.

Availability attack

Poisoning that broadly degrades model accuracy or usefulness, visible in aggregate metrics.

Integrity attack

Targeted poisoning that makes the model wrong only on specific attacker-chosen inputs while behaving normally elsewhere.

Backdoor

An integrity attack whose malicious behavior is gated by a trigger the attacker inserts at inference time.

Trigger

A specific input pattern that activates a backdoor's malicious behavior while the model stays clean without it.

Attack success rate (ASR)

The fraction of targeted or triggered inputs on which the poisoned behavior occurs.

Certified defense

A defense that gives a formal bound on the worst-case harm a bounded poison fraction can cause.

References

1. Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (arXiv 1708.06733)
2. Carlini et al., Poisoning Web-Scale Training Datasets Is Practical (arXiv 2302.10149)
3. Tran et al., Spectral Signatures in Backdoor Attacks (NeurIPS 2018)
4. Steinhardt et al., Certified Defenses for Data Poisoning Attacks (NeurIPS 2017, arXiv 1706.03691)
5. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)
6. SLSA: Supply-chain Levels for Software Artifacts