Evasion and Robustness: Mimicry Attacks on Behavioral Detectors and Their Provable Limits

Abstract

Behavioral defenses are necessary for non-deterministic agents but bounded by a classical result: an adversary who understands the detector can mimic normal behavior while achieving a malicious goal. This article formalizes evasion as a constrained reachability problem, catalogs the mimicry primitives an adaptive attacker uses — each paired with a countermeasure — and states the provable limit precisely: if any behavior the detector accepts also achieves the objective, no threshold prevents evasion; detection only raises cost. It then defines robustness as shrinking the accepted-yet-harmful region and increasing the constraints an attacker must satisfy at once, and insists on adaptive evaluation against an adversary who knows the defense. The conclusion is honest: behavioral defense is a cost-raising layer whose strongest partner is capability reduction, not a boundary that holds alone.

The uncomfortable fact at the center of behavioral security is that a detector announces, by its very design, what it considers normal — and anything an adversary can learn, an adversary can imitate. Two decades ago, Wagner and Soto showed that an intrusion detector built on sequences of system calls could be defeated by an attacker who padded a malicious sequence with innocuous ones until it looked ordinary. The same logic applies, unchanged, to an agent whose behavior is modeled and gated: if the attacker can reach the goal by a path the model accepts, the model does not stop the attack. This article takes that limit seriously — formalizes it, shows how it is exploited, and then asks what robustness can and cannot do about it.

The Adversary's Problem

Recast enforcement from the attacker's side. A behavioral detector partitions the space of possible agent behavior into an accepted region — behavior it scores as normal enough to allow — and a rejected region. The attacker's objective defines a separate set: the behaviors that achieve the malicious goal. The attacker does not need to enter the rejected region; the attacker needs only to find a behavior that lies in both the accepted region and the goal-achieving set at once.

This is a constrained optimization, not a brute-force one. The adversary is not trying random actions hoping to slip through; the adversary is solving a search — among all the ways to achieve the goal, find one the detector accepts, and among those, the one that is cheapest to construct. Framing evasion this way, as an informed search under a conformance constraint, is what separates the mimicry adversary from the noise the detector was tuned to catch.

The strength of the detector, from this view, is not how tightly it fits normal but how empty the intersection is. A detector that accepts a wide region gives the attacker room to maneuver; one that accepts only a narrow region forces the attacker into a harder search or none at all. Robustness, developed later, is precisely the project of shrinking that intersection.

\[\text{find } b \in \mathcal{A} \cap \mathcal{G}, \quad \mathcal{A} = \{ b : s(b) \le \tau \}\ (\text{accepted}),\ \mathcal{G} = \{ b : \text{goal}(b) \}\ (\text{achieves attack})\]

Mimicry, Formalized

Mimicry is the technique of shaping malicious behavior so that it falls inside the detector's accepted region. Wagner and Soto established its power against host-based intrusion detection: given knowledge of the accepted sequences, an attacker can transform an attack into a behaviorally-equivalent one that the detector passes, by substituting and padding operations while preserving the malicious effect. The result was not that a particular detector was weak but that a whole class of sequence detectors shared this vulnerability structurally.

The key move is separating what an action does from how it looks to the detector. If two different behaviors achieve the same malicious effect but one is accepted and one is rejected, the attacker simply uses the accepted one. The detector, which scores appearance and not effect, cannot tell them apart. Every behavioral detector that scores a proxy for maliciousness — order, frequency, embedding position — rather than the maliciousness itself inherits this gap between the proxy and the target.

Two adversary models bound the problem. A white-box attacker knows the detector and its parameters and can solve the conformance constraint directly. A black-box attacker does not, but can probe the detector with queries, observing which behaviors are flagged, and reconstruct enough of the accepted region to mimic it. Both are realistic for a deployed agent whose responses to actions are observable, which is why assuming the detector is secret is a weak foundation, not a strong one.

An attack succeeds against a behavioral detector exactly when a goal-achieving behavior is also one the detector accepts.

Attack Primitives and Their Countermeasures

Adaptive mimicry is built from a small set of primitives, each of which has a corresponding defense that does not eliminate it but raises its cost. Padding interleaves accepted, innocuous actions among the malicious ones so the sequence's statistics look normal; the countermeasure is to score effect-bearing actions with content signals — which arguments, which egress — so that padding the order does not launder the payload. Substitution replaces a flagged action with an accepted one that achieves the same end; the countermeasure is to constrain capability so that no accepted action achieves the harmful effect in the first place.

Low-and-slow spreads a malicious campaign across time and sessions to keep any window below threshold; the countermeasure is cumulative and cross-session accounting of irreversible effects, so that quantity over time is itself a monitored quantity. Query-based reconstruction probes a learned detector to map its boundary; the countermeasure is to limit and monitor probing, randomize or ensemble detectors so the boundary is a moving target, and never rely on a single static, queryable model. None of these defenses closes the gap; each makes the attacker's constrained search harder to solve.

The discipline this section models is the one the whole field should adopt: never present an attack primitive without its paired countermeasure, and never present a countermeasure as a cure. Each pairing is a cost increase, and security is the accumulation of cost increases the adversary must pay simultaneously, not any single decisive control.

An adaptive adversary composes padding, substitution, low-and-slow, and probing to satisfy the detector's conformance constraint.

The Provable Limit

The limit can be stated exactly. If the malicious goal is reachable by at least one behavior the detector accepts — if the intersection of the accepted region and the goal-achieving set is non-empty — then no choice of threshold on that detector prevents the attack, because the attacker will simply use a behavior from the intersection. Tightening the threshold shrinks the accepted region and may empty the intersection, but only at the price of rejecting more benign behavior; on a non-deterministic agent whose normal is broad, that price is a rising false-positive rate that eventually makes the detector unusable.

This yields a clean impossibility framing. A behavioral detector alone cannot both accept the full range of an agent's legitimate behavior and reject every behavior that achieves a malicious goal, whenever some legitimate-looking behavior can achieve that goal. The detector can only trade the two errors against each other along its operating curve. What it cannot do is make the intersection empty by tuning, because the intersection is a property of the agent's capabilities and the goal, not of the threshold.

The constructive reading is more useful than the pessimistic one. Since the intersection is set by capability, the way to empty it is to change capability: remove the agent's ability to achieve the harmful effect through any accepted action. That is not a detection move at all — it is capability reduction, and it is why the strongest partner of behavioral defense is least privilege on effects, not a better detector. Detection governs the residue that capability reduction cannot remove.

⚠️

Tuning cannot empty the intersection. If a malicious goal is reachable by any behavior the detector accepts, no threshold stops it — only removing the capability, or paying an unusable false-positive rate, can.

What Robustness Actually Buys

Robustness is the set of moves that shrink the accepted-yet-harmful intersection and raise the cost of searching whatever remains. The first and strongest is capability reduction: constrain what tools can do — least-privilege effects, strict egress allow-lists, quantity caps on irreversible actions — so that fewer goal-achieving behaviors exist to be mimicked at all. This attacks the intersection at its source rather than trying to detect entries into it.

The second is layering independent signals. Each additional signal the attacker must satisfy — order, argument content, egress destination, cumulative quantity, timing — adds a constraint to the mimicry search, and constraints that are statistically independent multiply the difficulty rather than adding to it. A mimic that conforms in order but must also conform in egress category, and in cumulative volume, and in argument shape, faces a search that can be made arbitrarily hard as independent signals accumulate. Correlated signals, by contrast, add little, so the design goal is independence, not quantity.

The third is denying the attacker a stable target. A single static, queryable detector can be reconstructed and then mimicked at leisure; an ensemble whose members are randomized or rotated, with some thresholds kept non-public, turns the boundary into a moving target and raises the query cost of black-box reconstruction. This is not security through obscurity as a primary defense — the layers above must hold even if the detector is known — but as a cost multiplier on top of them it is legitimate and worthwhile.

Each layer the mimic must satisfy simultaneously multiplies the difficulty of the evasion search.

Evaluating Robustness Honestly

The most common evaluation error is to measure a behavioral detector against a fixed set of non-adaptive attacks and report the resulting detection rate as robustness. That number is close to meaningless, because the real adversary is adaptive: it will observe the detector and solve the conformance constraint, so a detector's accuracy against attacks that did not adapt tells you almost nothing about its accuracy against attacks that do. Robustness is a property with respect to an adaptive adversary or it is not robustness.

A sound evaluation is adaptive. Give a red team knowledge of the detector and let it construct mimicry attacks — padding, substitution, low-and-slow, probing — then measure the cost it must pay to succeed: how many probes, how much padding, how much the campaign must be slowed, how much capability it must find unconstrained. The right metric is evasion cost and the size of the residual intersection, reported as a distribution against an adaptive attacker, not a single accuracy figure against a frozen test set.

This is the data-science discipline the topic demands: state the threat model, adopt the strongest adversary you can implement, measure effort rather than a point accuracy, and report what remains reachable. A robustness claim that does not name its adversary and its cost metric is not a claim about robustness — it is a claim about a test set the adversary has no reason to respect.

Non-adaptive versus adaptive evaluation of a behavioral detector.
Aspect	Non-adaptive (weak)	Adaptive (sound)
Adversary	Fixed attack set	Knows the detector, optimizes
Metric	Detection accuracy	Evasion cost + residual intersection
Signals tested	Often one	Independence of layered signals
Result meaning	Overstates robustness	Reflects real difficulty

Where Behavioral Defense Genuinely Helps

None of this argues against behavioral defense; it locates its value. Against an unsophisticated or non-adaptive adversary, a behavioral detector catches attacks that make no effort to conform, which is a large and worthwhile class. Against a sophisticated adaptive adversary, it raises the cost of every attack, forcing a constrained search that layering and capability reduction can make prohibitively expensive. What it never does is guarantee, alone, that a reachable malicious goal is unreachable — that guarantee comes only from removing the capability.

The right architecture therefore treats behavioral detection as the middle of a defense, not the whole of it. Capability reduction shrinks the harmful set toward empty; behavioral detection and graduated enforcement govern the residue that capability cannot remove; and designed reversibility ensures that the attacks which do slip through are recoverable. Each layer is bounded, and the security is in their composition, exactly as zero-trust and defense-in-depth have long held for systems that could never be made perfect at any single point.

The honest summary is the one worth carrying: behavioral security for agents is a genuine and necessary advance over static signatures and permissions, and it is a cost-raising layer with a provable ceiling. Design it to raise cost as much as possible, evaluate it against an adversary who knows it, and never let its presence substitute for shrinking what the agent can actually do.

Shrink the harmful-and-accepted set first, by constraining capability and effects.
Layer statistically independent signals so mimicry must satisfy many constraints at once.
Evaluate against an adaptive adversary and report evasion cost, not point accuracy.
Compose detection with reversibility so slip-throughs are recoverable, not catastrophic.

Key takeaways

Evasion is a constrained search: the attacker seeks a behavior that both achieves the goal and lies in the detector's accepted region, so a detector's strength is how empty that intersection is.
Mimicry works because detectors score a proxy (order, frequency, embedding) rather than effect; if two behaviors share a malicious effect but only one is accepted, the attacker uses the accepted one.
The provable limit: if any accepted behavior achieves the goal, no threshold prevents the attack — tuning only trades detection against a rising false-positive rate; it cannot empty the intersection.
Because the intersection is set by capability, the strongest defense is capability reduction (least-privilege effects, egress allow-lists, quantity caps), with detection governing the residue.
Robustness comes from layering statistically independent signals and denying a stable target; each independent constraint multiplies the difficulty of the mimicry search.
Evaluate robustness against an adaptive adversary that knows the detector and report evasion cost and residual reachability — accuracy against a frozen attack set overstates robustness.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

✅Adaptive robustness reviewchecklist

Confirm a behavioral defense was assessed against an adversary that knows it.

Capability is constrained so few goal-achieving behaviors exist (harmful set shrunk at the source).
At least three statistically independent signals must be satisfied at once (order, content, egress, quantity).
A red team with knowledge of the detector attempted padding, substitution, low-and-slow, and probing.
The reported metric is evasion cost and residual reachability, not accuracy against a fixed attack set.
Detectors are randomized or rotated and some thresholds are non-public as a cost multiplier, not a primary defense.
Slip-through actions are reversible, so a successful evasion is recoverable.

🔒Capability-reduction snippet (illustrative)policy

Shrink the harmful-and-accepted set directly, so there is less to mimic.

effects:
  egress:
    allow: [ "corp-crm.internal" ]     # explicit allow-list, deny by default
    deny_default: true
  irreversible_caps:                     # cumulative, cross-session
    send_email:   { per_hour: 5, per_day: 20 }
    delete_record:{ per_hour: 0 }        # requires human approval, never autonomous
  quantity_monitor: cumulative           # low-and-slow becomes a measured quantity
signals_required: [ order, arg_content, egress_category, cumulative_quantity ]  # independent

Least-privilege effects and cumulative caps

🧪Adaptive-evaluation loop (sanitized)harness

A no-op skeleton that measures evasion cost against a knowledgeable red team; not an attack tool.

function evaluate_robustness(detector, goal, budget):
    # red team KNOWS the detector (white-box); mock effects only
    cost = 0
    while cost < budget:
        candidate = red_team.construct_mimicry(detector, goal)  # padding/sub/slow/probe
        cost += candidate.effort                                 # probes + padding + delay
        if detector.accepts(candidate) and candidate.achieves(goal):
            return { "evaded": True, "evasion_cost": cost }
    return { "evaded": False, "evasion_cost": ">= budget" }        # report cost, not accuracy

Cost-based robustness measurement

Glossary

Mimicry attack: Shaping malicious behavior so it falls inside a detector's accepted region while still achieving the attacker's goal.
Accepted region: The set of behaviors a detector scores as normal enough to allow at its chosen threshold.
Evasion intersection: The overlap between the detector's accepted region and the behaviors that achieve the malicious goal; non-empty means the detector is evadable.
Padding: Interleaving innocuous, accepted actions among malicious ones to make a sequence's statistics look normal.
Substitution: Replacing a flagged action with an accepted action that achieves the same malicious effect.
Low-and-slow: Spreading a malicious campaign across time and sessions to keep any observation window below threshold.
Adaptive evaluation: Assessing a detector against an adversary that knows it and optimizes to evade, measuring evasion cost rather than accuracy against fixed attacks.

Abstract

The Adversary's Problem

Mimicry, Formalized

Attack Primitives and Their Countermeasures

The Provable Limit

What Robustness Actually Buys

Evaluating Robustness Honestly

Where Behavioral Defense Genuinely Helps

Key takeaways

Practitioner Toolkit

Glossary

References