Threat Modeling AI Agents by Capability and Provenance

Classic threat modeling fails on agents because its unit of analysis — the element on a data-flow diagram — cannot represent a threat that rides legitimate content across a boundary chosen at runtime. Criticism is cheap; the harder question is what to do instead. This piece answers it concretely. The move is to change the unit: stop asking what can go wrong at each box, and start maintaining an inventory of what the agent can reach and whose words are allowed to steer it. That inventory, made enforceable by a broker and gated in your pipeline, is a threat model an agent cannot walk out of unnoticed.

From critique to method

Two shifts carry the whole method. The first is provenance: for every span of tokens that enters the model's context, record where it came from and how much you trust it, and constrain what authority is reachable while a low-trust span is present. The second is capability: maintain a first-class inventory of the tools the agent can actually reach, each carrying the authority it borrows and the boundary it crosses. Together they answer the question STRIDE could not — not 'what can go wrong at this box' but 'whose words are steering the interpreter, and what can those words reach.'

Neither shift is exotic. Provenance is the security engineer's oldest instinct — trust nothing you did not originate — applied to a token stream. Capability inventory is the same discipline a post-quantum program uses for cryptography, or a zero trust architecture uses for access: enumerate the dependency, name its owner, verify every use. NIST's zero trust guidance (SP 800-207) is the closest existing frame: never trust a static perimeter; authorize each request against least privilege. An agent's tool calls are exactly such requests.

The rest of this piece is the mechanics: the inventory artifact, how to derive it, how provenance tags the context, how a broker enforces it, and how continuous integration keeps it honest.

The Agent Authority Inventory

The artifact at the center of the method is the Agent Authority Inventory: one row per capability the agent can actually reach, carrying the fields a classic threat model discards. It is deliberately shaped like a cryptographic bill of materials — an owned, reviewable list — so a security team can run it on a familiar cadence.

Each row answers four questions and records two more. What can this call do, and whose authority does it borrow? What boundary does that authority cross? Can untrusted content influence the deciding input — the injection surface — and does instruction share a channel with data? Then: who owns it, and what is its current exposure. The two columns that make this more than a permissions matrix are injection surface and data-control coupling, because they are the preconditions for the confused-deputy chains an agent is prone to.

The value of naming the columns is that blanks become findings. An unresolved 'authority' cell is a finding. 'Coupling: yes' with no compensating control is a finding. Two rows claiming the same privileged action is a contradiction worth investigating. The inventory turns 'we threat-modeled the agent' from a claim into a checkable state.

Deriving the inventory from a threat model

You do not start from a blank page; you derive the inventory from the threat model you already have, by asking a short set of low-level questions in the same session. For each data flow that touches the agent, refuse the checkbox and resolve it to parameters.

The pipeline is five steps. Enumerate every capability the agent can actually reach — not the documented tools, the reachable ones, including transitively through other tools and sub-agents. For each, resolve the concrete authority and the identity it borrows. Map the boundary that authority crosses. Mark whether untrusted content can influence the deciding input. Then gate the row and assign an owner: it is either an accepted risk with a control, or a finding someone owns. The single most common error is step one — enumerating documented capability instead of reachable capability, which undercounts the surface by construction.

The output is not a diagram you file away; it is a living list tied to the agent's tool manifest and regenerated whenever tools, scopes, model, or prompts change.

Provenance: making trust explicit per span

The inventory says what a capability can do; provenance decides who is allowed to trigger it. The mechanism is to tag every span of context entering the model with its source and a trust level, and to carry that tag through to the point where a tool call is authorized. Operator instructions are high trust; a user's message is medium; retrieved documents and tool outputs are low — data, not commands.

Two controls follow from the tags. First, spotlighting: mark low-trust spans so the model is instructed and structurally biased to treat them as inert, which reduces — though it does not eliminate — the chance a planted instruction is obeyed. Second, and more important because spotlighting is imperfect, the trust level becomes an input to authorization: a low-trust span may request information but must not, by itself, drive a high-authority tool that crosses an external boundary. Provenance is what lets the broker in the next section make that decision.

Recording trust as data rather than assuming it is the whole point. It converts an implicit hope — 'the model will know what to obey' — into an explicit, enforceable property of each span.

The broker: one checkpoint for authority

An inventory and provenance tags are inert until something enforces them. That something is a tool broker: a component outside the model that every tool call passes through, which authorizes the call against the inventory row and the provenance of the spans that produced it. Putting the check outside the model matters, because the model is the persuadable element — you cannot ask the thing being manipulated to also be the guard.

The broker's decision is concrete. Look up the capability's row. Deny by default. Require that the trust level driving the call meets the row's minimum — a low-trust span cannot invoke a high-authority tool alone. If the row crosses an external boundary, require human confirmation. Apply the row's egress allow-list so the destination is constrained even if the call fires. Issue a short-lived, audience-restricted token so the borrowed authority cannot be widened or replayed. Every one of those checks reads a field the inventory already carries.

This is the zero trust stance applied to agent tool use: the perimeter is not a wall the agent sits behind, it is a decision made per call, at a checkpoint the model does not control.

A worked example: turning a row into a finding

Take the support agent's send_email capability. Fill the row: authority is Mailbox.Send as a service identity; boundary is internal to external; injection surface is yes, because the agent reads retrieved documents; coupling is yes. Read across that row and it is not a description — it is a pre-computed attack path. Untrusted content can reach the deciding input, the tool holds real authority, and the impact crosses an external boundary. That is the confused deputy, written down before an attacker walks it.

The value is that the mitigation is now obvious and located. Because coupling is yes, break it with provenance and spotlighting so retrieved text cannot pose as an operator instruction. Because the boundary is external, the broker requires human confirmation. Because the authority is real, scope it to the minimum and issue a short-lived token. Because impact must be contained, attach an egress allow-list. The row does not just flag danger; it tells you exactly which of the four controls this capability is missing.

Run this for every row and the inventory becomes a map of your agent's exploitable paths and the specific control each one still needs — the opposite of a threat model that inspected each element and found nothing.

Gating it in continuous integration

An inventory that is not gated decays into documentation. The operational move is to make the Agent Authority Inventory a build-time artifact: a new tool, a widened scope, a changed system prompt, or a swapped model must add or update a row, and a row that crosses an external boundary with unmitigated coupling fails the build until an owner accepts the risk or attaches a control. This mirrors how mature programs gate on a cryptographic bill of materials — no undocumented dependency ships — and it inherits the same benefit: the artifact stays true because the pipeline enforces it.

Drift is the real adversary of the inventory's truth. An agent's reachable authority changes when the model, the prompt, the retrieved corpus, or the tool set changes — any of which can silently widen the surface. So the inventory is regenerated on those events, not annually, and the gate runs on every change. Ownership closes the loop: every row names an accountable team, because the risk usually lives in the seam — the agent team owns the prompt, the platform team owns the tool, and neither owns the authority the combination creates. NIST's AI Risk Management Framework frames this as mapping and governing risk to accountable roles; the gate operationalizes it.

The result is a threat model that is executable rather than aspirational: it runs in the same place your tests do, and it blocks the same way a failing test does.

1. Regenerate the inventory on any change to tools, scopes, model, or prompt.
2. Fail the build on any row with coupling = yes, an external boundary, and no compensating control.
3. Require an owner on every row; an unowned capability is a blocking finding.
4. Re-run the provenance and broker checks as tests, not as a one-time review.

What this catches that per-element modeling cannot

The method's power is that its unit of analysis can hold the agentic threat. Per-element enumeration inspects a box and asks which of six named threats apply; the confused-deputy path fires none of them, so it passes. A capability row asks a different question — can untrusted words reach this authority, and how far does that authority reach — and the same path lights up immediately as injection surface plus high authority plus an external boundary.

It also composes cleanly with what already works. Below the model, the broker, the token service, and the egress proxy are ordinary components with ordinary threats, and classic threat modeling handles them well. Use it there. The capability-and-provenance model is the addition at exactly the element where a model turns content into action — the one place per-element modeling is blind. A mature agent threat model runs both, and is honest about where each one's coverage ends.

None of this is a claim that the agentic threat is solved. Provenance is imperfect, models are non-deterministic, and a determined attacker will probe the seams. The claim is narrower and more useful: with authority and provenance as the unit, the threat is finally representable — you can write down the exploitable paths, assign each an owner and a control, and gate the whole thing — which is the precondition for defending it at all.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# illustrative, not runnable
capability: send_email
identity: svc-agent-mailer          # borrowed identity, not the user
scope: [Mailbox.Send]               # narrowest scope that works
boundary: external
injection_surface: true             # reads retrieved documents
authorize:
  min_trust: high                   # low-trust spans may NOT drive this call
  human_confirm: true               # because boundary == external
  egress_allow_list: ["*@corp.example"]
token: { ttl_seconds: 300, audience: mail.internal }
default: deny

# sanitized: static check over the inventory file
def gate(inventory):
    for row in inventory:
        risky = (row.injection_surface and row.boundary == "external"
                 and row.authority == "high")
        mitigated = (row.controls.spotlight and row.controls.broker
                     and row.controls.human_confirm and row.controls.egress_allow_list)
        if risky and not mitigated:
            fail(f"{row.capability}: unmitigated confused-deputy path")
        if row.owner is None:
            fail(f"{row.capability}: no owner")
    passed()

Glossary

Agent Authority Inventory

A living record of every capability an agent can reach, with the authority it borrows, the boundary it crosses, injection surface, coupling, owner, and exposure.

Provenance

The recorded source and trust level of a span of context, carried through to the point where a tool call is authorized.

Tool broker

A component outside the model that authorizes every tool call against the inventory row and the provenance of the spans that produced it.

Data-control coupling

The condition in which the instruction steering a decision can arrive on the same channel as untrusted data — the default in LLM agents.

Reachable capability

The set of tools an agent can invoke directly or transitively, as opposed to the smaller documented set.

Spotlighting

Marking untrusted context spans so the model is instructed and structurally biased to treat them as inert data.

References

1. OWASP Top 10 for LLM Applications (LLM01 Prompt Injection, LLM06 Excessive Agency)
2. OWASP Agentic Security Initiative — Agentic AI Threats and Mitigations
3. MITRE ATLAS — Adversarial Threat Landscape for AI Systems
4. NIST SP 800-207 — Zero Trust Architecture
5. NIST SP 800-154 — Guide to Data-Centric System Threat Modeling
6. NIST AI Risk Management Framework (NIST AI 100-1)
7. N. Hardy (1988), The Confused Deputy, ACM SIGOPS Operating Systems Review
8. Saltzer & Schroeder (1975), The Protection of Information in Computer Systems, Proc. IEEE
9. Greshake et al. (2023), Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv:2302.12173)