The Detection Threat Model: Evasion, Spoofing, and Scrubbing Goals

It is tempting to treat 'is this AI-generated?' as a classification problem: build a good detector, measure its accuracy, deploy it. But the moment a detector is used as a control — to gate academic submissions, flag disinformation, or enforce provenance — it acquires an adversary, and an adversary changes everything. A spam filter's accuracy on random email is irrelevant to its accuracy against a spammer trying to evade it; likewise, an AI-content detector's accuracy on honest text is irrelevant to its behavior against someone actively trying to fool it. The detection threat model exists to make this shift explicit: detection is not a measurement of content, it is a game against an opponent who gets to move after you, and its guarantees must be stated in those terms.

What detection promises, and to whom

AI-content detection comes in two broad forms. Watermarking embeds a statistical signal into generated content at generation time — Kirchenbauer and colleagues' green-list scheme biases token choices so a detector with the key can later recognize the signal — so detection is a matter of testing for that signal. Post-hoc classifiers, by contrast, infer AI-origin from statistical properties of content they did not generate, with no embedded signal to rely on. The two have very different security properties, and conflating them obscures what each can promise.

The crucial question is: promises to whom, against whom? A detector might be highly accurate on honest content — real AI text and real human text sampled naturally — and that accuracy is what benchmarks usually report. But a control's users are honest and its adversaries are not, so the promise that matters is what holds when someone is actively trying to defeat the detector. Honest-case accuracy and adversarial robustness are different quantities, and only the latter is a security property.

The defensive framing is that a detection control's guarantee must be stated against a specified adversary and scoped to what survives that adversary. Claiming a detector is '99% accurate' without naming the adversary is like claiming a lock is secure without saying against whom — meaningful for honest passers-by, meaningless against a lockpick. The threat model's job is to name the adversaries and their goals.

The adversary's three goals

The detection adversary is not monolithic; they have distinct goals that threaten different things. Evasion: make AI-generated content read as human, defeating detection to pass off AI text as one's own or to spread AI disinformation undetected. Spoofing: make content register as AI when it should not — either framing human-written text as AI, or forging a specific author's watermark to attribute content falsely to them. Scrubbing: remove a watermark from AI content so it can be redistributed as unmarked, defeating provenance.

These goals threaten different stakeholders. Evasion threatens anyone relying on detection to catch AI content — educators, platforms, fact-checkers. Spoofing threatens the accused: a human framed as an AI author, or an author framed for content they did not write, suffers a false positive weaponized against them. Scrubbing threatens provenance systems: content whose origin was marked becomes unmarkable, breaking attribution. A detector strong against one goal may be weak against another, so the threat model must consider all three.

The tree below decomposes the adversary's aims. Naming them separately matters because defenses differ: robustness to evasion is about the watermark surviving transformations, resistance to spoofing is about the watermark being unforgeable, and resistance to scrubbing is about the signal being hard to remove without destroying the content. A control that conflates them will be evaluated on the wrong axis.

Detection is an adversarial game

The defining feature of the detection threat model is that the adversary moves after the detector. The detector is fixed (or slowly updated); the adversary observes it and adapts. This ordering — defender commits, attacker responds — is the structure of an adversarial game, and it means a detector's static accuracy is the wrong metric. What matters is the adversary's best response: how well can someone who knows the detector defeat it? Sadasivan and colleagues argued, along these lines, that reliable detection of AI text faces fundamental difficulty precisely because a capable adversary can adapt.

This game structure has a sobering implication: as generative models improve, the gap between AI and human text narrows, which shrinks the signal a post-hoc classifier can exploit and makes evasion easier. A detector that works today may fail as models improve, not because the detector got worse but because its adversary got better. Security claims must therefore be time-scoped and adversary-scoped, not stated as permanent accuracy.

The defensive posture that follows is humility calibrated to the game. Watermarking has an advantage here because it embeds a signal rather than hoping one exists, but even a watermark faces an adaptive remover. So the realistic stance is to state what the detector guarantees against a defined adversary at a defined capability level, to expect that adversaries adapt, and to avoid deploying detection as a sole, high-stakes gate whose failure is catastrophic.

The false-positive stakes

Detection's error modes are not symmetric in their harm, and the threat model must weigh them. A false negative (AI content read as human) lets AI content through — the evasion adversary's win — which is bad but often recoverable. A false positive (human content flagged as AI) accuses an innocent author, and when detection gates consequential decisions (academic integrity, employment, publication), a false positive is a direct harm to a person. The spoofing adversary weaponizes exactly this: inducing false positives to frame someone.

This asymmetry means a detector's false-positive rate under adversarial conditions is a first-class security concern, not a minor accuracy footnote. A detector that can be made to flag arbitrary human text as AI is a tool for defamation, and a watermark that can be forged onto another author's content is a tool for false attribution. The threat model must ask not just 'can AI content evade?' but 'can human content be framed?' — and the second question is often the more damaging.

The defensive implication is to treat detection outputs, especially positive ones, as evidence to corroborate rather than verdicts to act on. Because adversarial false positives are achievable and their harm is high, a positive detection should trigger review and require corroboration before any consequential action, and the system should be designed so that a single detector's false positive — potentially attacker-induced — cannot by itself ruin someone.

Scoping honest deployment

Given the game, the honest way to deploy detection is with scoped, humble claims and corroboration. State the specific adversary the control is robust against and at what capability; do not claim robustness you cannot demonstrate against an adaptive attacker. Use detection as one signal among several, corroborated before consequential action, never as a sole high-stakes gate. And prefer, where provenance is the goal, cryptographic content-provenance approaches that make positive claims (this content was produced by this signer) over statistical detectors that make fragile negative inferences (this looks AI-generated).

The distinction between statistical detection and cryptographic provenance is important and recurs later in this series. Statistical detection asks an adversarial question about arbitrary content and is fundamentally gameable; cryptographic provenance attaches a verifiable signature at creation and answers a different, stronger question about content whose origin was recorded. Neither solves the other's problem, but understanding which question a system actually needs — 'is this AI?' versus 'who made this?' — determines which tool is appropriate and how much to trust it.

The synthesis, grounded in Kirchenbauer, Sadasivan, and Kuditipudi and their colleagues, is that AI-content detection is a security control facing motivated adversaries with distinct goals — evasion, spoofing, scrubbing — so it must be evaluated adversarially, scoped in its claims, and deployed with corroboration. Honest-case accuracy is not security; the security of a detector is what survives an attacker who moves second, and stating detection's promises in those terms is the foundation of using it responsibly.

Why this matters for agents and platforms

Platforms increasingly want to detect AI-generated content — for moderation, provenance, or policy — and agents increasingly generate content at scale, making detection both more desired and more strained. A platform that deploys a detector as a hard gate inherits all three adversaries: bad actors evading to spread AI content, malicious actors spoofing to frame others, and redistributors scrubbing watermarks. Treating the detector's benchmark accuracy as its security guarantee sets up exactly the failures the threat model predicts.

The organizing lesson is that detection controls must be built and evaluated as adversarial systems: name the adversary, measure best-response robustness, weigh asymmetric error harms, and deploy with corroboration rather than as a sole verdict. Kirchenbauer's watermarking, Sadasivan's detectability limits, and the broader literature agree that reliable adversarial detection is hard, so responsible deployment scopes its claims and never lets a gameable detector make an irreversible decision about a person. For platforms and agents, the question 'is this AI?' is genuinely useful only when asked with the adversary in mind.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

detection_control_policy:
  claims:
    state_adversary: required
    state_capability_level: required
    unscoped_accuracy_claims: forbidden
  evaluation:
    metric: adversarial_best_response
    goals_considered: [evasion, spoofing, scrubbing]
  errors:
    adversarial_false_positive: first_class_concern
  use:
    role: one_corroborated_signal
    sole_high_stakes_gate: forbidden
    positive_requires_corroboration: true
  provenance:
    prefer_cryptographic_for_who_made_this: true

DETECTION CONTROL DISCLOSURE:
  Detector type: [watermark | post-hoc classifier]
  Robust against: [named adversary] at [capability level]
  NOT robust against: [known evasion/spoofing/scrubbing gaps]
  Honest-case accuracy: [x] (NOT a security guarantee)
  Adversarial best-response accuracy: [y]
  Error harms: false positive can [frame a person]; corroboration required.
  Deployment: one signal among several; never a sole high-stakes gate.

Glossary

AI-content detection

Determining whether content is AI-generated, via embedded watermarks or post-hoc statistical classifiers.

Watermarking

Embedding a statistical signal into generated content at generation time so a detector with the key can recognize it.

Post-hoc classifier

A detector that infers AI-origin from statistical properties of content it did not generate, with no embedded signal.

Evasion

Making AI-generated content read as human to defeat detection.

Spoofing

Making content register as AI when it should not, including forging another author's watermark for false attribution.

Scrubbing

Removing a watermark from AI content so it can be redistributed as unmarked, defeating provenance.

Adversarial robustness

A detector's accuracy against an attacker who knows and adapts to it, the true security property.

Content provenance

Cryptographically attaching verifiable origin to content at creation, answering 'who made this?' rather than 'is this AI?'.

References

1. Kirchenbauer et al., A Watermark for Large Language Models (arXiv 2301.10226)
2. Sadasivan et al., Can AI-Generated Text Be Reliably Detected? (arXiv 2303.11156)
3. Kuditipudi et al., Robust Distortion-Free Watermarks for Language Models (arXiv 2307.15593)
4. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology
5. C2PA (Coalition for Content Provenance and Authenticity) technical specification
6. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)