RAG and Fine-Tune Poisoning: Dose-Response of Corrupting an Index or Instruction Set

There is a tempting assumption that poisoning is a frontier-lab problem — that it takes control of a web-scale crawl to bend a model. It does not. The datasets most teams actually control are small: a retrieval index of company documents, a fine-tuning set of a few thousand instruction examples. Precisely because they are small and sit right against the application, a few poisoned items are a large fraction, and a large fraction is a reliable effect. An attacker who can add a handful of documents to a RAG store or slip a few examples into a fine-tune set is often better positioned than one straining to influence a billion-example pretraining run, because their dose lands closer to the behavior they want to change.

Two surfaces, two failure modes

Retrieval-augmented generation and fine-tuning poison the model at different places and persist differently. Fine-tune poisoning corrupts the training examples used to specialize a base model; the poison is learned into the weights and travels with the model wherever it goes, exactly like the pretraining backdoors but on a smaller, cheaper dataset. RAG poisoning corrupts the retrieval index the model reads at inference; the poison is not in the weights at all but in the documents the model is given as context, so it acts at retrieval time and can be added, changed, or removed after deployment.

This distinction matters for both attacker and defender. Fine-tune poison is durable and stealthy — once baked in, it needs no ongoing access — but requires influencing the training set before the run. RAG poison is dynamic and requires ongoing write access to the index, but it is trivially updatable and does not require any training at all; the attacker simply ensures the malicious document is retrievable for the queries they care about. One is a training-time integrity attack; the other is a runtime content-injection attack that rides the retrieval channel.

The unifying observation is that both are dose-response problems on small, application-proximate datasets, so both are defended by the same discipline — provenance and vetting scaled to the dataset's leverage, not its size.

The dose-response relationship

The central quantitative question is dose-response: how does downstream behavior change as the number of poisoned items grows? Across poisoning studies the shape is consistent — targeted effect rises steeply with the poisoned fraction and saturates, so there is a threshold dose above which the attack is reliable and below which it is not. On a small dataset, that threshold corresponds to a small absolute count, which is exactly why application-proximate surfaces are cheap to attack.

For fine-tuning, the dose is the number of poisoned examples relative to the fine-tune set size; a set of a few thousand examples can be swung by a poisoned count in the tens. For RAG, the dose is subtler: what matters is not the poison's fraction of the whole index but its dominance of the neighborhood for the targeted queries — a poisoned document that reliably lands in the top-k for a query class controls the context for that class regardless of the index's total size. In both cases the effective dose is local, not global.

The defender's job is to raise the required dose and the cost of each poisoned item. Deduplication and diversity in retrieval prevent a few near-identical poison documents from dominating a neighborhood; per-source contribution caps limit how much any contributor can add to a fine-tune set; and provenance makes each poisoned item traceable and rejectable. Each pushes the attacker toward a larger, more detectable dose.

Poisoning the retrieval index

RAG poisoning is a runtime attack on the retrieval channel. The attacker crafts a document that (a) will be retrieved for the queries they care about and (b) carries the malicious content — a false fact, a hidden instruction, or a biased framing — that the model will then incorporate into its answer. Getting retrieved is a similarity-optimization problem: the document is written or tuned so its embedding sits near the target query region, so it reliably enters the top-k. This is content-level poisoning that rides the same retrieval mechanics a legitimate document uses.

Because the index is updated continuously and often ingests documents from untrusted or semi-trusted sources — user uploads, crawled pages, connected data stores — the write surface is broad and the poison is dynamic. An attacker can insert, refine, and remove poison to evade detection, and because RAG systems treat retrieved content as authoritative context, a retrieved poison document is effectively an instruction or fact the model trusts.

The defenses are the retrieval-security controls applied with poisoning in mind: authenticate the provenance of every ingested document, cap how much any one source can place near a query region, deduplicate to prevent neighborhood domination, and — critically — treat retrieved content as untrusted data that informs rather than commands, so a poisoned document cannot directly drive a consequential action.

Poisoning the fine-tune set

Fine-tune poisoning targets the small, high-leverage instruction or preference dataset used to specialize a model. Because these sets are assembled from community contributions, purchased data, synthetic generation, or user interactions, an attacker with any contribution path can insert poisoned examples that teach a targeted behavior — a backdoor trigger, a systematic bias, or a specific false response. The small size of the set means a modest number of poisoned examples is a meaningful fraction, and instruction-tuning's strong shaping of behavior makes each example high-impact.

The stealth of fine-tune poison is that, once learned, it is indistinguishable from legitimately learned behavior by inspection of the weights alone, and the poisoned examples themselves may be individually plausible — especially in a clean-label construction. Standard evaluation on a general benchmark will not surface a targeted or triggered behavior, so the poison survives to deployment and travels with the model into every downstream use.

Defenses combine source vetting (know and authenticate who contributed each example), dataset signing (so the fine-tune set cannot be altered between assembly and training), deduplication, and targeted plus trigger-aware testing before deployment. Because the set is small, careful human and automated review of contributions is actually feasible here, unlike at web scale — which makes rigorous vetting the highest-leverage control for this surface.

# DEFENSIVE GATE — vet contributions before they enter a fine-tune set
function vet_contribution(example, source):
    if not authenticated(source): reject("unauthenticated source")
    if source.fraction_so_far > PER_SOURCE_CAP: reject("source cap")
    if near_duplicate(example, accepted_set): drop("dedup")
    if label_content_inconsistent(example): flag("dirty-label review")
    record_provenance(example, source)          # sign into the set
    return accept(example)

# Pre-deploy: run targeted + trigger-aware tests on the fine-tuned model,
# not only aggregate benchmarks.

Why proximity beats scale

The through-line of both surfaces is that proximity to the application beats raw scale. A pretraining poison must survive dilution across a vast corpus and the averaging of a huge training run; a fine-tune or RAG poison acts on a small, focused dataset whose every item strongly shapes a narrow behavior. The closer the poisoned data sits to the deployed decision, the smaller the dose needed and the more precisely the attacker can aim, which inverts the intuition that bigger training pipelines are the bigger risk.

This reframes where defensive effort should go. Teams often assume their own small datasets are too minor to attract poisoning and reserve rigor for large training runs, but the economics point the other way: the small, proximate dataset is the cheaper and more reliable target. Provenance, vetting, deduplication, and targeted testing on fine-tune sets and retrieval indexes are therefore not optional hygiene but primary controls, and their small size makes thorough application of those controls achievable.

The synthesis, consistent with the poisoning literature and OWASP's flagging of vector and embedding weaknesses, is that dose-response plus proximity make application-proximate datasets a high-value, low-cost target — so defend them with provenance and vetting at least as strict as anything applied upstream.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE PROBE — is one source dominating retrieval for a query class?
function domination_probe(index, canary_queries, k):
    for q in canary_queries:
        results = index.search(q, k)
        by_source = tally(r.source for r in results)
        top_share = max(by_source.values()) / k
        if top_share > DOMINATION_THRESHOLD:
            flag(q, source=argmax(by_source), share=top_share)
    # High single-source share == possible index poisoning; investigate.

proximate_data_policy:
  retrieval_index:
    ingestion_provenance: required
    write_access: controlled
    per_source_region_cap: 0.1
    dedup: true
    retrieved_content_trust: untrusted_data
  fine_tune_set:
    source_vetting: required
    per_source_cap: 0.05
    dataset_signing: required
    dedup: true
  pre_deploy_tests:
    targeted_cases: required
    trigger_aware: true

Glossary

Retrieval-augmented generation (RAG)

An architecture where a model retrieves documents from an index and uses them as context to answer a query.

Fine-tuning

Specializing a base model on a smaller task-specific dataset, learning its patterns into the weights.

Dose-response

The relationship between the amount of poison injected and the size of the resulting behavioral effect.

Local dose

The poison amount relative to the small, relevant subset (a fine-tune set or a query neighborhood) rather than the whole corpus.

Index poisoning

Inserting documents into a retrieval index so they are retrieved for targeted queries and inject malicious content.

Instruction-tuning set

A dataset of instruction-response examples used to shape a model's behavior during fine-tuning.

Dataset signing

Cryptographically binding an assembled dataset so it cannot be altered between assembly and training.

Neighborhood domination

A poison document reliably occupying the top-k results for a query class, controlling the retrieved context.

References

1. Carlini et al., Poisoning Web-Scale Training Datasets Is Practical (arXiv 2302.10149)
2. Gu et al., BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain (arXiv 1708.06733)
3. Steinhardt et al., Certified Defenses for Data Poisoning Attacks (NeurIPS 2017, arXiv 1706.03691)
4. Tran et al., Spectral Signatures in Backdoor Attacks (NeurIPS 2018)
5. OWASP Top 10 for LLM Applications (LLM08 Vector and Embedding Weaknesses)
6. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)