The Agent Threat Model — AI Red-Teaming & Agent Security

A conventional application waits for a user to click a button or call an application programming interface (API). An agent can be asked for an outcome, choose steps, call tools, read data, write data, and try again when something fails. That extra freedom is useful, but it changes the security question. You are no longer only asking, “Can this system protect data?” You are asking, “What could this system decide to do, who could influence that decision, and how far could the consequences travel?”

Start with what an agent is

An agent is software that uses a model, usually a large language model (LLM), to decide what to do next in pursuit of a goal. It may read a message, inspect files, search a knowledge base, call a tool, update a ticket, send an email, or ask a human for approval. The important feature is not that it speaks in natural language. The important feature is that it selects actions.

A threat model is a structured way to ask what can go wrong before it goes wrong. For an agent, the threat model begins with a simple loop: observe, decide, act, and learn from the result. Each pass through that loop can improve the outcome, but each pass can also compound a mistake.

📌

The key shift. A chatbot mainly produces text, while an agent can produce changes in other systems.

An agent repeats observe, decide, act, and learn; each pass can improve the outcome or compound a mistake.

The first question is not about prompts

Prompt attacks get attention because they are visible. Someone tells the model to ignore its instructions, reveal a secret, or take a harmful action. These attacks matter, but they are not the whole threat model. A prompt is only one way to influence an agent.

From first principles, security starts with three questions: what assets matter, who can influence the system, and what actions are possible. An asset may be customer data, source code, a payment approval, a calendar, a privileged credential, or the reputation attached to an official message. An influence may come from a user, a web page, an email, a document, a search result, a plug-in, a tool response, or another agent.

The dangerous pattern is indirect control. The agent may trust content that was never meant to be trusted. A support email can contain instructions. A web page can contain hidden text. A document can include a command disguised as data. If the agent treats all text as instructions, the boundary between user intent and untrusted content collapses.

Treat user goals, system policy, tool output, and outside content as different kinds of input.
Record which inputs are trusted, which are untrusted, and which are allowed to affect actions.
Assume hostile instructions can arrive inside ordinary business data.

Draw the trust boundaries

A trust boundary is a line where the system must stop assuming that data, identity, or intent is safe. Agents cross many of these lines. They may move from a chat message into a document store, from a document into an enterprise system, from a summary into an email draft, and from a draft into a sent message.

The useful exercise is to describe the agent’s world in plain nouns and verbs. What can it read? What can it write? What can it delete? What can it send outside the organization? What can it purchase, approve, unlock, or publish? For each verb, ask whether the agent is allowed to do it alone, allowed to propose it, or forbidden to do it.

This is where many weak designs become visible. A single tool called “run_query” may allow both safe lookup and destructive update. A broad service account may let the agent read every mailbox when it only needs one shared queue. A search tool may blend public web content with confidential internal records without marking the difference.

⚠️

Verb test. If a tool name hides whether it reads, writes, sends, deletes, or approves, it is too vague for a good threat model.

Model the main failure modes

Agent failures are often combinations, not single bugs. A malicious document may steer the model. The model may choose an overpowered tool. The tool may use a broad identity. The result may be sent to the wrong person. None of these pieces looks dramatic alone, but together they form an incident.

Prompt injection is the attempt to override or manipulate the agent’s instructions through text the model processes. Data exfiltration is the movement of information to a place where it should not go. Tool misuse is the agent’s use of a legitimate capability in an unsafe way. Excessive agency is when the agent has more freedom than the task requires. Confused deputy behavior occurs when an attacker tricks a more privileged system into doing something on the attacker’s behalf.

A sound threat model names these patterns without assuming the model will reliably refuse them. Language models are good at interpreting text, but that same strength makes them vulnerable to persuasive, hidden, or conflicting instructions. The defense is not to find a magic sentence that makes the model safe. The defense is to constrain the system around the model.

Instruction override: untrusted content tells the agent to ignore higher-priority rules.
Secret spill: the agent includes confidential data in a response, ticket, log, or external message.
Unsafe action: the agent takes a real-world step without enough certainty or approval.
Scope creep: a tool or identity grants access beyond the task.
Feedback poisoning: the agent learns from or relies on manipulated results.

Agent failures cluster by who acts and what is at risk: attacker input or agent action, data or control.

Put identity and permission below the model

The model should not be the only thing deciding what the agent can do. Identity and permission systems should enforce limits even when the model is confused, manipulated, or wrong. If the agent is not allowed to delete records, the delete operation should fail regardless of how confidently the model asks for it.

Use least privilege: give the agent only the permissions needed for its specific job, for the shortest practical time, on the narrowest practical data set. In a Microsoft environment, that means using Microsoft Entra ID to assign and govern identities, permissions, conditional access, and audit trails. In any environment, it means avoiding broad shared secrets, long-lived credentials, and tools that bypass normal access checks.

Separate user authority from agent authority. Sometimes an agent should act as the user, inheriting that user’s access. Sometimes it should act as a service with its own limited role. Both patterns can be valid, but mixing them casually creates ambiguity. The audit log should show who requested the action, which agent performed it, which tool was used, and what permission allowed it.

✅

Fail closed. A safe design makes the dangerous action fail outside the model, not merely sound discouraged inside the prompt.

Identity and permission enforce limits under the model, so a fooled model still cannot exceed its rights.

Design tools as safety boundaries

Tools are where an agent’s words become effects. A weather lookup tool is low risk. A refund tool, code deployment tool, payment tool, or customer messaging tool is higher risk because it changes records, money, software, or relationships. The more serious the effect, the more precise the tool contract must be.

A good tool is narrow, typed, logged, and policy-aware. Narrow means it does one kind of action. Typed means its inputs are structured, such as an amount, account identifier, or date, rather than an open-ended instruction string. Logged means each call can be investigated later. Policy-aware means the tool checks limits itself, such as maximum refund amount, allowed recipient domain, or required approval state.

Human approval is not a cure-all, but it is useful at high-consequence points. The approval must show the human the actual action, not just a friendly summary. “Approve refund of 42.00 dollars to customer 12345” is reviewable. “Approve the agent’s plan” is not.

Prefer many small tools over one powerful general tool.
Require structured parameters instead of free-form commands.
Validate tool inputs against business policy before execution.
Add approval gates for actions that send, spend, delete, publish, or grant access.
Log the prompt context, selected tool, parameters, result, and approver where policy permits.

Tools sit on a risk gradient from read-only lookups to actions that move money or change systems.

Turn the threat model into operating decisions

Leaders and practitioners eventually face tradeoffs. More autonomy can reduce manual work, but it also increases the blast radius of a mistake. More data can improve answers, but it increases exposure if retrieval or output controls fail. More integrations can make the agent useful, but each integration adds a new place where authority, data, or intent can be misused.

The practical decision is not “Are agents safe?” The practical decision is “Which tasks can this agent perform under which controls?” A low-risk internal summarizer may need content filtering, access checks, and logging. A sales assistant that drafts emails may need recipient controls and human review before sending. An operations agent that can restart services or change configuration needs tight identity, approval gates, rollback plans, and incident monitoring.

Platforms such as Microsoft Foundry can help teams build, evaluate, and operate agents, but the platform does not remove the need to decide risk appetite. A vendor can provide guardrails, evaluation tools, identity integration, and monitoring hooks. The organization still owns the choice of what the agent may do, what evidence is required before launch, and when a human must stay in the loop.

📌

Decision rule. Increase autonomy only when you can also increase evidence, constraint, monitoring, and accountability.

Test the agent like an adaptive system

Traditional testing asks whether known inputs produce expected outputs. Agent testing must also ask whether unexpected inputs lead to unsafe choices. The test set should include normal requests, ambiguous requests, malicious instructions, poisoned documents, tool failures, permission failures, and attempts to extract secrets.

Evaluation should happen before release and continuously after release. Models change, tools change, data changes, and attackers adapt. Monitor not only final answers but also intermediate steps: which sources the agent used, which tools it selected, which actions failed, which approvals were requested, and which policies blocked execution.

The goal is not to prove the agent can never fail. The goal is to make failures less likely, less severe, easier to detect, and easier to recover from. A mature threat model ends with operational habits: review logs, rotate credentials, investigate anomalies, update tests from real incidents, and keep a clear kill switch for disabling risky capabilities.

Define the agent’s allowed tasks and forbidden tasks.
Map assets, inputs, tools, identities, and trust boundaries.
Create abuse cases for prompt injection, data leakage, unsafe actions, and excessive permission.
Add technical controls outside the model wherever possible.
Test with hostile and ambiguous inputs, then monitor real behavior after launch.

⚠️

No final prompt. There is no single instruction that can replace least privilege, tool design, approval gates, logging, and incident response.

Key takeaways

An agent threat model focuses on what the system can see, decide, and do.
Untrusted content can influence an agent indirectly through emails, documents, web pages, tool results, and other data.
The strongest controls sit outside the model: identity, permissions, tool limits, approvals, and logging.
Tools should be narrow, structured, policy-aware, and auditable because they turn model decisions into real effects.
Higher autonomy requires stronger evidence, tighter constraints, better monitoring, and clearer accountability.
Agent testing must include hostile inputs, unsafe tool paths, permission failures, and continuous monitoring after launch.

Glossary

Agent: Software that uses a model to choose and perform steps toward a goal, often by calling tools or using data.
Threat model: A structured description of what can go wrong, what matters most, and which controls reduce the risk.
Large language model: A model trained to process and generate language, often used by agents to interpret requests and choose next steps.
Prompt injection: An attempt to manipulate a model by placing hostile or misleading instructions in text the model reads.
Trust boundary: A point where data, identity, or authority moves from one level of trust to another and must be checked.
Least privilege: The practice of granting only the minimum access needed to perform a specific task.
Data exfiltration: The unauthorized movement of information to a person, system, or location that should not receive it.
Confused deputy: A failure where a less trusted actor tricks a more privileged system into using its authority for the wrong purpose.