Layered Outcome Assurance: An Architecture for Securing Agents That Act

Picture an agent that has been reviewed with care. It may read customer records, because it answers account questions. It may create access for a new user, because it helps with onboarding. It may send a message outside the organization, because it follows up with partners. Each permission was justified, documented, and approved. Then a shared document arrives containing one hidden sentence, and the agent reads the customer file, grants an unknown account access, and mails the file to an outside address. No component failed. No single call was unauthorized. The organization still lost its customer data and gained a back door that nobody ever approved. That gap between approved parts and an unapproved whole is the problem this architecture exists to close.

The component fallacy

Start with definitions, because the argument depends on them. An agent, in this article, is a software system that uses a language model to interpret a goal, choose actions, and invoke tools that change the world, repeating that loop until it judges the goal complete. A tool is any callable capability the agent can use, such as a database query, an identity service, or an email sender. A principal is the person or system on whose authority an action is taken. A privileged action is any tool call whose effect matters if it is wrong: it moves data, changes access, spends money, or cannot easily be undone.

Traditional security review is componential. It asks whether each piece is secure: is this identity scoped, is this tool authenticated, is this call authorized, is this model filtered. For software that follows a fixed path written by a developer, that question is close to sufficient, because the path itself was reviewed. An agent does not follow a fixed path. It composes its path at run time, from content it reads and plans it forms, so the review of the parts no longer implies anything about the whole. We call the belief that it does the component fallacy.

The fallacy is not hypothetical. Greshake and colleagues demonstrated in 2023 that instructions hidden in retrieved content, such as a web page or an email, can take control of real applications built on language models, a class of attack now called indirect prompt injection. The model does not need to be broken for this to work. It only needs to treat part of the content it reads as an instruction, which is exactly what a capable, helpful model tends to do.

The deeper pattern is older than language models. In 1988, Hardy described the confused deputy: a program that holds legitimate authority is tricked into exercising it on behalf of a party who holds none. An agent with several broad, individually justified permissions is a confused deputy waiting for instructions, and every document it reads is a candidate to supply them. The attacker never acquires a permission. The attacker borrows the agent's.

The running example in this article is exactly such a deputy. Three permissions, each approved, compose into data loss plus a quiet back door, triggered by untrusted content. Any architecture that claims to secure agents has to stop that outcome without depending on the model noticing the trick, and it has to explain afterward what happened and why. The rest of this article builds one that does.

From components to outcomes

If the unit of review cannot be the component, it has to be the outcome. An outcome is a change in the world that someone cares about: customer data leaving the organization, a new account holding access, a payment being made. Outcomes are what a business approves or forbids. Components are merely how outcomes happen. An assurance architecture for agents should therefore specify, in advance, which outcomes an agent may reach, and provide mechanisms that keep every other outcome unreachable regardless of what the model decides.

We use the word guarantee for a property that an architecture makes true by construction rather than by hoping the model behaves. A guarantee is stated as a sentence that can be tested: for example, content that did not come from a trusted principal cannot choose the recipient of an outbound message. A layer is a group of mechanisms that together make exactly one guarantee true. A contract is what a layer requires from the layers beneath it and what it supplies to the layers above it. Conformance evidence is a test that shows the guarantee holds in a running system.

From those definitions, five requirements follow, and they shaped every choice in the architecture. First, it must be outcome-centred: its guarantees mention outcomes and the paths to them, not only permissions. Second, it must be model-independent: no guarantee may depend on the language model detecting an attack, because the model is the component an attacker addresses directly. Third, it must be compositional: the guarantees must fit together so that each covers a gap the others leave. Fourth, it must be testable: each guarantee comes with conformance evidence. Fifth, it must degrade gracefully: losing one layer should weaken the system, not collapse it silently.

None of these ideas is new in isolation, and it would be dishonest to present them as new. Saltzer and Schroeder set out the design principles for protecting information in 1975, including least privilege, fail-safe defaults, complete mediation of every access, and economy of mechanism. What changes with agents is where those principles must be applied. The thing that needs mediation is no longer a single access request from a known program; it is a chain of actions composed at run time by a system that reads adversarial content. Layered Outcome Assurance is our synthesis of the classic principles into a structure fitted to that new unit of risk.

The six layers and their guarantees

Layered Outcome Assurance has six layers. Each is named for what it secures and defined by one guarantee. The layers are ordered from the conditions the system must assume at the base to the decisions it enforces at the top.

Layer 1, Continuous Pressure. Guarantee: the system's security does not depend on attacks being rare, slow, or manual. Public threat assessments, such as the 2024 assessment by the United Kingdom's National Cyber Security Centre, expect artificial intelligence to raise the volume and impact of cyber attacks over the near term, mainly by making reconnaissance and social engineering cheaper and faster. When each attempt costs an attacker little, an attacker can afford to keep trying until a probabilistic defence fails once. Layer 1 turns that into design assumptions every other layer must honour: controls are sized for volume, not for rarity, and a defence that works most of the time is treated as a defence that will eventually fail.

Layer 2, Agents as Actors. Guarantee: every agent is a known actor whose reach is bounded and can be enumerated. Reach is the full set of outcomes an agent could cause, determined by four things together: the identity it acts with, the information it can see, the tools it can call, and the sequences of calls it can make. Least privilege applied to each of these separately is necessary but not sufficient; the guarantee is about their combination, stated as an inventory the upper layers can reason over.

Layer 3, The Action Chain. Guarantee: content that did not come from a trusted principal cannot determine which privileged action runs or what arguments it receives. This is information-flow control applied to agents. Denning showed in 1976 how labels attached to data can be propagated through computation so that information from one class cannot flow into a place reserved for another. Debenedetti and colleagues applied a related idea directly to agents in 2025: their system, CaMeL, separates the plan derived from the trusted user request from the untrusted data the agent processes, and tracks where each value came from so that untrusted data cannot redirect privileged calls.

Layer 4, Whole Behavior. Guarantee: no permitted sequence of actions reaches an outcome that nobody approved. Permissions are evaluated one call at a time; outcomes are produced by sequences. Brewer and Nash formalised a history-based policy in 1989 in which what a subject may access next depends on what it has already accessed, to prevent conflicts of interest. Layer 4 generalises that idea to agents: the decision about the next action takes into account the actions already taken in the same task, so that combinations can be approved or forbidden as combinations.

Layer 5, Explanatory Evidence. Guarantee: every consequential action leaves a record sufficient to explain why it happened and who is accountable, and that record resists tampering. A conventional log says that an operation was called and succeeded. An explanatory record says which inputs arrived and from where, what was decided and under which policy, which identity acted on whose behalf, with which parameters, who approved it, and what changed. Crosby and Wallach showed in 2009 how logs can be made tamper-evident, so that an insider or an attacker cannot silently rewrite history.

Layer 6, Deterministic Enforcement. Guarantee: the decision whether a consequential action executes is made by a mechanism outside the model that returns the same verdict for the same inputs. The model proposes; an independent mechanism decides. Expressing that mechanism as policy written in a language designed to be analyzed, such as the Cedar authorization language described by Cutler and colleagues in 2024, makes it possible not only to run the decision but to test and reason about it before it is deployed.

The dependency contract

A stack of layers is only an architecture if the layers depend on one another in defined ways. In Layered Outcome Assurance, each layer consumes something produced beneath it and supplies something that the layers above cannot produce for themselves. We call the full set of these obligations the dependency contract. The dependencies are informational, not merely a build order: a layer can be implemented at any time, but it can only deliver its guarantee once it receives what the contract says it needs.

Layer 1 supplies the threat assumptions that every other layer designs against, most importantly that attempts are numerous and persistent. Layer 2 consumes those assumptions and supplies the reach inventory: for each agent, the identities, information, tools, and reachable sequences. Without that inventory, Layer 3 cannot know which calls are privileged and therefore which arguments must be protected, and Layer 4 cannot know which combinations of actions are even possible.

Layer 3 supplies provenance: labels on data and arguments that record whether each value came from a trusted principal or from untrusted content. Layer 4 consumes provenance and reach, and supplies two things upward: the set of approved outcomes expressed as rules over sequences, and a statement of which history must be remembered for those rules to be evaluated. Layer 5 consumes that statement and supplies the evidence record, which is also the history that Layer 4's rules need at decision time.

Layer 6 consumes rules from Layer 4, provenance from Layer 3, reach from Layer 2, and history from Layer 5, and supplies verdicts: allow, deny, or escalate to a human or a stricter check. Every verdict is itself written into the Layer 5 record, and patterns in those records, such as repeated denied attempts, inform the Layer 1 assumptions. The contract is therefore a loop through its top and bottom, not a one-way stack, and it is the loop that lets the architecture learn from the pressure it is under.

The contract also makes failure analysable. If a layer receives less than the contract promises, its guarantee weakens in a predictable way, and the weakness can be named before an incident names it. That property matters more than any single mechanism, because every mechanism will eventually be bypassed, misconfigured, or outgrown.

What fails when a layer is skipped

The most useful property of a layered architecture is that each layer's absence has a distinct, recognisable signature. Naming those signatures in advance turns an abstract diagram into a review tool: an assessor can look at an agent, find the missing layer, and predict the incident it invites.

Skip Layer 1, and the other layers are designed for an attacker who gives up. Filters are tuned so that they rarely block legitimate content, alerts are tuned so that they rarely fire, and the design quietly assumes that a manipulation which fails most of the time is safe. Against an attacker for whom each attempt is nearly free, a defence that fails occasionally is a defence that will be defeated, merely later.

Skip Layer 2, and nobody can say what the agent could do. The organisation knows each permission but not their product, so the path from reading customer records to sending them outside is invisible until it is used. Every upper layer then works from guesses about which calls matter. Skip Layer 3, and the agent's plan can be rewritten by whatever it reads. A document, a web page, a tool description, or a retrieved record can supply the recipient, the file, or the account that the next privileged call acts on.

Skip Layer 4, and each call is judged alone. This is the component fallacy expressed as a missing layer: read allowed, grant allowed, send allowed, outcome never evaluated. Skip Layer 5, and the organisation learns about the incident without being able to explain it. The logs show three successful calls and nothing about the document that caused them, the policy that was consulted, or the person on whose behalf the agent acted. Stateful rules in Layer 4 also lose their memory, because the history they need was never kept.

Skip Layer 6, and the final decision is made by the model, perhaps with a human approval step attached. The model can be persuaded; that is the premise of prompt injection. A human approver shown only an Approve button and no context is closer to a formality than a control. Without an independent, deterministic decision point, every other layer can advise, but nothing can refuse.

The running example, traced through all six layers

Return to the agent and the shared document, and follow the attack upward through a system that implements every layer. The purpose is not to show that one layer is enough, but that the layers overlap, so that the failure of any one of them is caught by another.

At Layer 1, the design already assumes that documents like this one will arrive repeatedly and in many variants, so no control relies on the injection being unusual or on the model spotting it. At Layer 2, the reach inventory for this agent shows a path from customer records to an external recipient, and from identity creation to a new account. That path is flagged as a candidate outcome requiring explicit approval, before any document arrives.

At Layer 3, the hidden sentence is data from an untrusted source, so every value derived from it carries an untrusted provenance label. When the plan tries to call the mail tool with a recipient and attachment that trace to that label, the arguments are rejected as untrusted, regardless of how persuasive the sentence was to the model. At Layer 4, even if the provenance labels were somehow lost, the sequence of reading customer data, creating access, and sending externally within one task matches a forbidden combination, and the next call in that sequence requires escalation.

At Layer 5, each attempted call is recorded with its inputs and their provenance, the rule consulted, the identity and principal, and the verdict, in a tamper-evident log. At Layer 6, the decision to execute the external send is made by an independent policy mechanism that receives the reach, the provenance, the rule, and the history, and returns a deny. The same inputs would produce the same deny tomorrow, which is precisely what makes the outcome dependable.

Three separate layers would each have stopped this outcome on their own, and two more would have explained it. That overlap is deliberate. It is what allows each mechanism to be imperfect, which every real mechanism is, without the architecture being imperfect in the same place at the same time.

Relationship to prior work

Layered Outcome Assurance draws on a long line of security research, and its value lies in the composition rather than in any single element. The design principles of Saltzer and Schroeder supply its vocabulary of least privilege, complete mediation, and fail-safe defaults. Hardy's confused deputy supplies its central threat: authority exercised on behalf of the wrong party. Denning's lattice model of information flow grounds the provenance guarantee of Layer 3, and Brewer and Nash's history-based policy grounds the sequence guarantee of Layer 4. Crosby and Wallach's tamper-evident logging grounds the integrity half of Layer 5.

Among agent-specific work, the demonstration of indirect prompt injection by Greshake and colleagues established the threat that Layers 3 and 4 answer, and CaMeL by Debenedetti and colleagues showed that separating trusted control flow from untrusted data can defeat many such attacks by design rather than by detection. Analyzable policy languages such as Cedar, described by Cutler and colleagues, show that authorization decisions can be both expressive and amenable to automated reasoning, which Layer 6 relies on.

At the governance level, the NIST Artificial Intelligence Risk Management Framework organises risk activity into functions for governing, mapping, measuring, and managing AI risk. Layered Outcome Assurance is complementary rather than competing: it is a technical architecture whose conformance evidence can feed those functions, supplying concrete answers to the questions that governance asks about what an agent can do, what it did, and why.

What is new here is the unit of analysis and the way the pieces are bound together: an outcome-centred set of guarantees, one per layer, joined by an explicit contract of what each layer needs and supplies, with a named failure signature when any layer is absent. Prior controls are reused deliberately; we claim the structure, not the parts.

Conformance: how to know a layer holds

A guarantee that cannot be tested is an aspiration. For each layer, Layered Outcome Assurance pairs the guarantee with conformance evidence that a team can produce in a test environment and repeat after every change.

For Layer 1, the evidence is volume testing: the same manipulation, varied and repeated many times, never produces a privileged outcome, rather than producing one rarely. For Layer 2, it is a complete reach inventory that can be regenerated from configuration and compared against what the agent is observed doing, with any observed action outside the inventory treated as a defect. For Layer 3, it is canary content: untrusted inputs carrying planted instructions that try to set the arguments of privileged calls, with the test passing only if no such argument ever originates from them.

For Layer 4, the evidence is a suite of forbidden sequences, each exercised end to end, each required to end in denial or escalation. For Layer 5, it is reconstruction: given only the evidence record of a test incident, an independent reviewer can say what happened, why, on whose behalf, and whether it can be undone, and any attempt to alter an earlier record is detected. For Layer 6, it is determinism and analysis: identical inputs produce identical verdicts across repeated runs, and the policy is checked automatically for properties such as never allowing an external send of customer data without approval.

• L1: repeated, varied manipulation never succeeds, rather than rarely succeeding
• L2: reach inventory regenerated from configuration and matched against observed behavior
• L3: canary content never supplies an argument to a privileged call
• L4: every forbidden sequence ends in denial or escalation
• L5: an incident is reconstructable from the record alone, and tampering is detected
• L6: same inputs yield the same verdict, and the policy passes automated property checks

Limitations and threats to validity

Layered Outcome Assurance is an architecture proposed from established principles and an analysis of known attacks; it has not yet been measured across a population of deployed agents, and its value should be judged by that standard as evidence accumulates. Several limitations are inherent and deserve to be stated plainly.

Every guarantee is only as strong as its mediation is complete. If an agent can reach a tool through a path that bypasses the Layer 6 decision, such as a direct network call or an unmonitored plug-in, the guarantee does not hold for that path, and no amount of policy elsewhere will restore it. Specifying approved outcomes is also genuinely hard. Rules that are too narrow block legitimate work and train people to request exceptions; rules that are too broad approve the very combinations they were meant to forbid.

Provenance tracking has known costs and limits. Labels can be lost when data passes through components that do not preserve them, and strict separation of plan and data can reduce what an agent is able to accomplish, which is a real trade-off rather than a free improvement. Evidence records contain sensitive information themselves, so they need their own access control and retention limits, and they add storage and latency. Escalation to humans is only a control when reviewers see real context and are not overwhelmed; too many escalations create approval fatigue and erode the check they were meant to provide.

Finally, model independence does not make the model irrelevant. A better model still produces fewer unsafe proposals and a better experience, and the architecture assumes the model is useful. What it refuses to assume is that the model is the security boundary. Multi-agent systems that cross organisational boundaries raise further questions of whose policy and whose evidence apply, which this architecture frames but does not resolve.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

Glossary

Agent

A software system that uses a language model to interpret a goal, choose actions, and invoke tools that change the world, repeating until it judges the goal complete.

Principal

The person or system on whose authority an action is taken.

Privileged action

A tool call whose effect matters if it is wrong, such as moving data, changing access, spending money, or making an irreversible change.

Outcome

A change in the world that someone cares about and would approve or forbid, such as customer data leaving the organization.

Guarantee

A property that an architecture makes true by construction, stated as a sentence that can be tested.

Layered Outcome Assurance

An architecture of six layers for agents that act, each making one guarantee true and bound to the others by a dependency contract.

Dependency contract

The set of obligations stating what each layer requires from beneath it and supplies to the layers above.

Reach

The full set of outcomes an agent could cause, determined by its identity, the information it can see, the tools it can call, and the sequences it can make.

Provenance label

A tag attached to a piece of data or an argument recording whether it came from a trusted principal or from untrusted content.

Indirect prompt injection

An attack in which instructions hidden in content an agent reads, rather than in the user's request, steer what the agent does.

Confused deputy

A program holding legitimate authority that is tricked into using it on behalf of a party that holds none (Hardy, 1988).

Complete mediation

The principle that every access to every protected object is checked, every time (Saltzer and Schroeder, 1975).

Conformance evidence

A repeatable test showing that a layer's guarantee holds in a running system.

Approval fatigue

The erosion of a human check when reviewers see too many requests with too little context to judge them.

References

1. Saltzer and Schroeder, The Protection of Information in Computer Systems (Proceedings of the IEEE, 1975)
2. Hardy, The Confused Deputy (ACM SIGOPS Operating Systems Review, 1988)
3. Denning, A Lattice Model of Secure Information Flow (Communications of the ACM, 1976)
4. Brewer and Nash, The Chinese Wall Security Policy (IEEE Symposium on Security and Privacy, 1989)
5. Greshake et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023)
6. Debenedetti et al., Defeating Prompt Injections by Design (CaMeL, 2025)
7. Cutler et al., Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization (2024)
8. Crosby and Wallach, Efficient Data Structures for Tamper-Evident Logging (USENIX Security, 2009)
9. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0), 2023
10. UK National Cyber Security Centre, The near-term impact of AI on the cyber threat (2024)