From Signatures to Behavior: Why Static and Permission Controls Fail on Non-Deterministic Agents

A permission system answers one question: is this identity allowed to perform this operation? For fifty years that question was enough, because the operations a program could perform were fixed when it was written. An autonomous agent breaks the premise. It decides at runtime which tools to call, in which order, with which arguments, and it may reach the same goal by different routes on different runs. The operation is authorized and the request is well-formed, yet the sequence is one no designer anticipated and no policy explicitly blessed. The failure is not that the old controls are implemented poorly; it is that they answer a question that no longer determines safety.

The Enumeration Assumption

Both dominant control paradigms share a hidden premise. A signature control — the mechanism behind classical antivirus and intrusion detection — accepts input by default and blocks anything matching a catalog of known-bad patterns. A static authorization control inverts the polarity: it blocks by default and permits only operations on an explicit allow-list, the discipline of least privilege. One enumerates the forbidden and one enumerates the permitted, but both assume the relevant set is knowable and stable at design time. Call this the enumeration assumption: that legitimate and illegitimate behavior can be separated by a fixed predicate written before the system runs.

The assumption held because traditional software is, in the relevant sense, deterministic and finite. A compiled program exposes a fixed set of entry points; a service exposes a fixed API surface; a user's role maps to a fixed set of permissions. The space of possible actions is enumerable, so a designer can reason about it exhaustively, and an auditor can check the enumeration for completeness. Least privilege, as articulated by Saltzer and Schroeder and later codified across security engineering, is precisely the practice of making that enumeration as tight as the function allows.

An autonomous agent is a program whose action sequence is not written in advance but generated at inference time by a language model conditioned on a goal, a context window, and the returned results of its own prior actions. The set of sequences it can produce is not enumerated anywhere; it is an emergent property of the model, the prompt, and the environment. The enumeration assumption does not merely strain under this workload — it is false, and every control built on it inherits a blind spot that no amount of tuning closes.

Why Signatures Fail on Generated Behavior

A signature detects a specific realization of an attack: a byte pattern, a system-call sequence, a request shape known to be malicious. It works when adversarial behavior recurs in a recognizable form, so that one observed instance protects against the next. Against an agent, the adversary's realized behavior is generated, not reused. A prompt-injection payload that redirects an agent toward exfiltration (catalogued by the OWASP Top 10 for Large Language Model Applications as a top-ranked risk) need not share any surface feature with the next such payload; the model can be steered to the same malicious end through unlimited paraphrases and tool orderings.

The MITRE ATLAS knowledge base, which adapts the ATT&CK methodology to artificial-intelligence systems, documents attack techniques — such as prompt injection and model evasion — whose defining property is that they exploit the model's generality rather than a fixed implementation flaw. A signature can encode yesterday's realization of such a technique; it cannot encode the technique's generative source. The gap between a technique and its instances is exactly where signatures lose their protective leverage.

There is a deeper, information-theoretic point. A signature is a compression of past malicious observations into a matcher. Its value depends on the entropy of the attack distribution being low enough that a bounded catalog covers most future instances. For agent behavior the attack distribution is high-entropy by construction, because the same objective admits astronomically many action sequences. A catalog that would cover it is not smaller than the space it describes, which defeats the point of a signature.

Why Static Permissions Fail on Intent

Static authorization fails for a different and subtler reason: it is intent-blind. A permission grants an identity the right to perform an operation, but an agent's danger lives not in any single operation it is entitled to perform, but in the purpose and combination of operations. Reading a document, querying a database, and sending an email may each be individually authorized, yet their composition — read a confidential file, encode it, mail it out — is exfiltration. No per-operation permission distinguishes the benign composition from the malicious one, because the permission layer never sees intent.

This is the confused-deputy problem in modern dress, and the OWASP Agentic Security Initiative names it directly under excessive agency: an agent granted broad tool access can be induced to wield that access toward an attacker's goal, with every individual call passing authorization. The pressure to over-grant is structural. An agent is useful in proportion to the breadth of tools it can reach, so operators face a standing incentive to widen the allow-list, and each widening enlarges the set of harmful compositions that remain individually authorized.

NIST Special Publication 800-207 anticipated the general failure when it defined zero-trust architecture around continuous, per-request evaluation rather than a one-time grant of standing privilege. Its tenet is that access decisions must be re-computed dynamically from observed signals, not read from a static entitlement. A permission list is precisely the standing-privilege model that zero trust was formulated to replace; applying it to agents reintroduces the assumption the architecture was built to reject.

The Non-Determinism That Breaks Both

Non-determinism is the property that makes the two failures inescapable rather than merely inconvenient. A language-model agent samples its next token from a probability distribution, and that stochasticity propagates: the same goal under the same context can yield different plans, different tool orderings, and different arguments on different runs. Tool-return timing and content vary as well, feeding back into subsequent decisions. The observable behavior is therefore a draw from a distribution over action sequences, not a fixed program trace.

Formally, let the agent's behavior on a task be a random variable over sequences, with support that is neither known in advance nor stable across runs. A signature control checks membership in a fixed bad set; a permission control checks membership in a fixed allowed set. Neither predicate can be correct for a target whose legitimate support is itself a distribution the designer never enumerated. What is legitimate is defined by the shape of that distribution, and a point predicate cannot express a shape.

This reframes the design problem. The right question is not is this action in the allowed set but is this action sequence consistent with how this agent normally behaves in this context. That is a question about a distribution, and answering it requires estimating the distribution from observation rather than declaring it by fiat. It is the question that behavior-based security exists to answer.

Behavior-Based Security, Defined

Behavior-based security replaces the fixed predicate with an estimated model of normal behavior and flags deviation from it. It is the security specialization of anomaly detection, whose canonical taxonomy — set out by Chandola, Banerjee, and Kumar — distinguishes point anomalies (a single instance far from the norm), contextual anomalies (normal in general but abnormal in this context), and collective anomalies (a subsequence abnormal as a whole though each element is ordinary). Agent misuse spans all three: a single forbidden tool call is a point anomaly, a normal call at an abnormal step is contextual, and a benign-looking read-encode-send chain is collective.

Concretely, one estimates a model of the agent's normal action distribution and assigns each observed behavior an anomaly score equal to its negative log-likelihood under that model; behavior whose score exceeds a threshold is flagged. The same taxonomy also fixes the supervision regime: labeled attacks are scarce and non-recurring, so the practical setting is semi-supervised — learn the shape of normal from benign operation, then treat improbable behavior as suspect — rather than supervised classification of known attacks.

The crucial shift is from enumeration to estimation. A signature or permission is authored; a behavioral model is fit. That difference is what lets the approach cover behavior no designer anticipated, because the model generalizes over the distribution rather than memorizing points in it. It is also the source of the approach's own weaknesses — estimation error, distribution shift, and the cold-start problem of having no normal to learn from — which any honest treatment must carry alongside the benefits.

The Hard Limit: Mimicry

Behavior-based defense has a classical and sobering limit, established by Wagner and Soto for host-based intrusion detection and fully applicable here: a mimicry attack. If an adversary knows, or can approximate, the model of normal behavior, it can craft malicious activity that stays inside the region the detector considers normal — achieving its goal while producing a behavior sequence indistinguishable, to the detector, from legitimate operation. The stronger and more public the behavioral model, the more precisely an adversary can shape its actions to evade it.

This does not refute behavior-based security; it bounds it. Mimicry raises the adversary's cost — the attacker must now solve a constrained problem, achieving the objective while conforming to a learned distribution — but it denies any claim of completeness. The correct posture is to treat every detection design together with its evasion analysis: for each behavioral signal, ask how an informed adversary would mimic normalcy, and layer independent signals so that conforming to all of them at once is harder than conforming to any one.

The lesson generalizes the article's thesis. There is no single predicate, static or learned, that separates safe from unsafe agent behavior for all adversaries. Static controls fail against generativity and non-determinism; behavioral controls fail against an informed mimic. Defense is therefore a layered, cost-raising program measured by how much adversary effort it forces, not a boundary that promises to hold.

Where It Fits: Continuous Verification for Agents

Behavior-based security is not a replacement for identity and least privilege but the missing runtime layer above them. Identity answers who; least privilege bounds what is reachable; behavioral conformance answers whether the reachable actions are being used as they normally are. This is the layering that zero-trust architecture already prescribes in general form: NIST Special Publication 800-207 makes access a continuous function of observed signals evaluated at a policy decision point, and an agent's behavioral conformance is exactly such a signal. The behavioral layer supplies the dynamic input that turns a static entitlement into a continuous decision.

The mapping to governance is direct. The NIST AI Risk Management Framework organizes practice into functions including MEASURE and MANAGE; a behavioral baseline is a MEASURE instrument (it defines and applies a metric of conduct), and flagging plus response is MANAGE (it acts on the measured risk). The Generative AI Profile that accompanies the framework names behavioral risks — confabulation, harmful action, data leakage — that are observable only as conduct, not as a static attribute, which is precisely what a behavioral layer is positioned to watch.

The consequence for design is a change of primitive. Instead of asking operators to enumerate every safe action in advance, a behavioral system asks them to observe what safe operation looks like and to treat departure from it as the trigger for scrutiny. The remaining engineering — how to build the baseline, how to score deviation despite scarce labels and shifting behavior, how to enforce on the resulting signal, and how to withstand a mimic — is the substance of behavior-based defense, and each part carries its own limits that a serious program states plainly.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

function deviation_score(trace, model):
    # model was fit on BENIGN traces only (semi-supervised)
    s = 0
    for step in trace:                       # step = (tool, arg_hash, context)
        p = model.likelihood(step.tool, step.context)   # P(action | context)
        s += -log(max(p, 1e-9))              # negative log-likelihood
    collective = model.sequence_likelihood(trace)        # collective-anomaly term
    return s - log(max(collective, 1e-9))

function decide(trace, model, tau):
    return "flag" if deviation_score(trace, model) > tau else "allow"

Glossary

Enumeration assumption

The premise, shared by signature and permission controls, that legitimate or illegitimate behavior can be separated by a fixed predicate written before the system runs.

Signature control

A default-allow mechanism that blocks inputs matching a catalog of known-bad patterns.

Least privilege

The discipline of granting an identity only the minimum operations its function requires, enumerated in advance.

Confused deputy

A privileged component induced to misuse its own legitimate authority on behalf of an attacker who lacks that authority.

Anomaly score

A numeric measure of how improbable an observed behavior is under a model of normal behavior, typically its negative log-likelihood.

Mimicry attack

An evasion in which an adversary who knows the detector's model of normal crafts malicious behavior that stays inside the normal region.

Policy decision point

In zero-trust architecture, the component that computes an access decision dynamically from observed signals for each request.

References

1. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
2. NIST AI 600-1, Generative AI Profile
3. NIST SP 800-207, Zero Trust Architecture
4. OWASP Top 10 for Large Language Model Applications (2025)
5. OWASP Agentic Security Initiative — Agentic AI Threats and Mitigations
6. MITRE ATLAS — Adversarial Threat Landscape for AI Systems
7. Chandola, Banerjee & Kumar, Anomaly Detection: A Survey (ACM Computing Surveys, 2009)
8. Wagner & Soto, Mimicry Attacks on Host-Based Intrusion Detection Systems (ACM CCS, 2002)