The Perceive-Act Loop: Why a Screen-Driven Agent Turns the Display Into a Trust Boundary

A traditional software agent calls tools through a typed interface: it invokes a named function with structured arguments, and the surface it can touch is fixed by a schema. A computer-use agent works differently and far more dangerously. It looks at a screen — a screenshot, an accessibility tree, or both — decides what to do, and then clicks and types like a person, over whatever software happens to be in front of it. That shift sounds like a convenience, but it quietly relocates the most important boundary in the system. The screen the agent reads is not a trusted readout of the agent's own intentions; it is rendered by the environment, and the environment can be a web page, a document, a desktop application, or a pop-up that an attacker influenced. The moment the agent's next action is a function of what is on the display, the display becomes an input — and an input that the agent implicitly trusts, cannot fully authenticate, and immediately acts upon. This article does not catalog specific screen attacks; it builds the map that later analysis needs: what the loop is, why the display is untrusted, and exactly where the trust boundary cuts through the perceive-act cycle.

What a Computer-Use Agent Actually Is

A computer-use agent is a system that operates general-purpose software the way a human does: it perceives the interface through pixels or a structured accessibility representation, and it acts through the same input channels a person uses — pointer clicks, keystrokes, scrolls. The distinguishing property is not intelligence but interface: rather than calling a curated set of application programming interfaces, the agent drives the graphical user interface itself, which means its action space is whatever the software on screen exposes to a user.

This capability class has a concrete research lineage. WebArena, introduced by Zhou, Xu, Zhu and colleagues, built realistic, reproducible websites so agents could be measured on long-horizon tasks a person routinely performs online, and reported that a strong contemporary agent completed only about a seventh of those tasks where humans completed roughly three-quarters — evidence that acting through a real interface is hard. OSWorld, from Xie, Zhang, Chen and colleagues, went further to full operating systems across Ubuntu, Windows, and macOS, and found that the best evaluated agent solved a small fraction of tasks that humans solved comfortably, with grounding identified as a primary bottleneck. These are capability benchmarks, not security studies, but they establish the substrate this article reasons about: agents that read a screen and act on arbitrary software.

The reason to define the class precisely is that its security properties follow from its interface, not its intelligence. Whatever the model's reasoning quality, a system whose next action is determined by the current display inherits the trust properties of that display — and those properties are the subject of everything below.

The Loop, Stage by Stage

The agent runs a repeating cycle with five distinguishable stages, and naming them precisely is what makes the later security argument tractable. Perceive is the capture of the current interface state — a screenshot, an accessibility tree, or a fusion of both — into the model's context. Ground is the translation of an intended target ('the submit button', 'the address field') into a concrete, actionable location: a specific element or set of coordinates. Decide is the choice of the next action given the perceived state and the task. Act is the emission of that action into the real interface — a click at a point, a typed string. Observe is the capture of the resulting new screen, which feeds the next perception.

Grounding deserves emphasis because it is both the hardest stage and the most security-relevant. Zheng, Gou, Kil and colleagues showed, in their SeeAct study, that a capable multimodal model could plan web tasks well yet frequently failed to convert a correct plan into a correct on-screen action, with grounding — not planning — the dominant source of error. OSWorld reported the same pattern at operating-system scale. Grounding is where an abstract intention becomes a physical act, so it is precisely where an ambiguity, an accident, or an adversary can redirect the agent to the wrong element while every other stage looks nominal.

The loop is a control loop with the environment inside it. Unlike a pure planner that reasons over a fixed internal state, the computer-use agent's state at every step is re-derived from an external display it does not author. That single fact — the environment sits inside the perception stage of the loop — is the seed of the trust problem.

Why the Display Is Untrusted Input

The central claim is simple to state and easy to under-appreciate: for a computer-use agent, the screen is input, not output. A human treats a display as a trusted rendering because the human authored the intent and reads the screen only to confirm it. The agent inverts this — it derives its intent from the screen — so the display functions as an incoming data channel. And that channel is filled by the environment: a website's DOM and pixels, a document's contents, an application's chrome, an operating-system notification, an injected pop-up. None of those are under the agent operator's control.

This is the same collapse of the data-instruction boundary that Greshake, Abdelnabi, Mishra and colleagues identified for text, where content retrieved into a language model's context could act as instructions rather than mere data, letting a remote party steer an application without any direct interface. Computer-use agents generalize that failure from text tokens to interface elements: anything the environment can put on screen is content the agent will perceive, ground against, and potentially act upon. The payload is no longer only words in a document; it is buttons, labels, overlays, and layout — the interface itself as a carrier.

Crucially, the agent has no reliable intrinsic way to authenticate the display. A legitimate button and an adversarial one are both just rendered elements; a real dialog and a spoofed one occupy the same pixel plane. Where a well-designed API surface carries provenance and typing, the screen is an undifferentiated visual field in which trustworthy and hostile elements are, by construction, indistinguishable to perception alone.

The Trust Boundary Runs Through the Loop

Because the environment sits inside perception and the consequences land at action, the trust boundary is not around the agent — it runs through the loop. Untrusted content enters at perceive. It is then carried, unlabeled, through ground and decide, where the agent's own reasoning laundishes it into a concrete, legitimate-looking action. The action is realized at act, where it becomes a real effect on real software. Observe closes the loop, but it too reads from the same untrusted display, so the feedback that might catch a problem is itself supplied by the potential adversary.

This locates the two stages that matter most for control. Perceive is the crossing: it is where hostile content becomes part of the agent's state, and therefore the natural place for provenance and isolation. Act is the realization: it is where an intent — honest or hijacked — becomes irreversible, and therefore the natural place for gating, confirmation, and reversibility. The middle stages, ground and decide, are exactly where you cannot place your trust, because by the time content reaches them it has already shed its origin and looks like the agent's own plan.

Framing it this way yields the article's organizing artifact, an Action-Loop Trust Boundary map: untrusted-in at perceive, laundering through ground and decide, irreversible-out at act, and a feedback channel at observe that shares the untrusted source. A defender who internalizes this map stops trying to make the middle of the loop trustworthy and instead hardens the two ends where control is actually possible.

A Threat Model for the Action Loop

At the loop level, the attacker's goal is compact: cause the agent to take an attacker-chosen action while every stage appears to function normally. Three broad primitives serve that goal, and this article treats them at the level of the map rather than specific mechanics. The first is to control what appears on screen — to place content in the environment the agent will perceive. The second is to exploit grounding, steering the agent to act on an element other than the one its plan intended, taking advantage of the fragility that the capability benchmarks already documented. The third is to corrupt the feedback, shaping what the agent observes so a wrong or malicious action looks successful and the loop proceeds without alarm.

These primitives compose into a chain that mirrors the loop itself: influence perception, bend grounding, and suppress the observation that would reveal the deviation. The impact set is whatever the software in view can do, which is the crux of why this modality is uniquely severe and is developed further below. The point of the threat model here is not to enumerate exploits but to show that the attacker's moves map one-to-one onto the loop's stages, which is why defenses must be positioned by stage.

This loop-level threat model is deliberately mechanism-agnostic. Specific techniques — decoy and overlay elements against grounding, blast-radius abuse at action, and hostile applications and pop-ups as environment — are their own studies. The contribution here is the target map they all attack: the same perceive-ground-decide-act-observe cycle, entered at perception and realized at action.

Why This Loop Is Harder to Secure Than a Text Agent

Two structural properties make the computer-use loop more dangerous than a text-tool agent. The first is an unbounded action space. A tool-calling agent can only invoke the functions in its schema, so its blast radius is enumerable and can be reasoned about in advance. A computer-use agent can operate whatever software is on screen, which OSWorld demonstrated spans real desktop applications, file input and output, and multi-application workflows. The consequence set is therefore the union of everything the reachable software can do — deleting files, sending messages, moving money, changing settings — and it is not enumerable from a schema.

The second property is that grounding is unreliable, and unreliability is itself a security problem. When agents mis-ground even without an adversary — the dominant error mode in both the web and operating-system benchmarks — a mistaken click and a hijacked click are observationally identical. There is no clean baseline of 'correct behavior' against which a malicious deviation stands out, because the honest behavior already includes frequent wrong actions. An attacker who nudges grounding is hiding inside the system's existing error distribution.

Together these mean the modality combines the largest possible consequence set with the weakest possible guarantee that an action matches intent. That is the opposite of what a security engineer wants, and it is why a text-agent mindset — validate the tool arguments, allow-list the functions — does not transfer directly. The equivalent controls must operate on a raw, untyped interface where 'the arguments' are pixels and coordinates.

Reading the Loop as a Security Engineer

The map turns into a design discipline once its two control points are taken seriously. At perception, the goal is provenance and isolation: know which parties can render into the agent's view, weight trusted interface chrome above arbitrary content, and shrink the environment so the 'who can put content on screen' set is small and known rather than the open world. This is where the untrusted input is caught, before it has been laundered into a plan.

At action, the goal is gating and reversibility: make irreversible, high-consequence actions require an extra gate; prefer actions that can be undone; and constrain the action types available for a given task so that the agent's raw ability to click anything is narrowed to what the task legitimately needs. This is where a hijack, if it slipped through perception, is denied its final realization. The middle of the loop is explicitly not a control point — the design accepts that ground and decide will faithfully carry whatever perception admitted.

Observation is the subtle third consideration. Because the feedback that would reveal a bad action is read from the same untrusted display, a defender should seek at least one independent signal — an out-of-band check that an action had its intended effect — rather than trusting the environment's own report. The loop's honesty cannot be self-certified from inside the loop.

What This Means for the AI-Agent Stack

Computer-use is the highest-blast-radius agent modality precisely because it discards the protective typing of a tool interface and acts through the same raw surface a human uses. For anyone assembling an agent stack, that reframes where the security budget should go: not primarily into the model's reasoning, which sits in the untrustworthy middle of the loop, but into the perception and action boundaries where control is actually exercisable. The OWASP Agentic Security Initiative and the broader agentic threat guidance point the same way — excessive agency and untrusted inputs are the load-bearing risks — and the perceive-act loop is where both concentrate at once.

The design stance that follows is portable to any screen-driven agent regardless of vendor or model. Assume the display is hostile until scoped otherwise. Put provenance and isolation at perception. Put gating and reversibility at action. Treat grounding uncertainty on a sensitive target as a stop, not a guess. And never let the loop certify its own success from feedback the environment authored. These are not five separate features; they are the direct consequences of one fact — that a computer-use agent turned its display from an output it reads into an input it obeys.

The single idea to carry forward is the boundary itself. A screen-driven agent's trust boundary runs through its own loop, entered at perception and realized at action, and every serious defense for this class is a decision about how to control those two crossings.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

task: fill-expense-form
allowed_apps: [expense-portal]        # no arbitrary software
allowed_actions: [click, type, scroll] # no downloads, no shell
irreversible_actions: [submit]         # requires confirmation gate
egress_allowlist: [expense-portal.internal]
reversibility: prefer_undoable
stop_conditions:
  - low_grounding_confidence on sensitive target
  - element_outside allowed_apps

# DEFENSIVE intent-match check around the ground->act transition.
planned = plan.target_description        # what decide intended
grounded = ground(screen, planned)       # concrete element chosen
if not matches(grounded, expected_targets_for(task)):
    halt('off-plan target')              # possible grounding hijack
if is_irreversible(action_for(grounded)):
    require_confirmation()               # gate at the action end
log(planned, grounded, confidence)       # out-of-band audit trail

Glossary

Computer-use agent

An agent that perceives a graphical interface and acts through human input channels (clicks, keystrokes) over general-purpose software.

Perceive-act loop

The repeating cycle of perceive, ground, decide, act, and observe that a screen-driven agent runs against its environment.

Grounding

The stage that translates an intended target into a concrete, actionable on-screen element or coordinate.

Trust boundary

The line separating untrusted, environment-authored content from the agent's trusted decision and action context.

Action space

The set of effects an agent can produce; for a computer-use agent it is whatever the reachable software exposes to a user.

Blast radius

The scope of consequences a single agent action can cause across the reachable software and data.

References

1. Greshake, Abdelnabi, Mishra, Endres, Holz & Fritz, Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2302.12173, 2023)
2. Zhou, Xu, Zhu et al., WebArena: A Realistic Web Environment for Building Autonomous Agents (arXiv 2307.13854, ICLR 2024)
3. Zheng, Gou, Kil, Sun & Su, GPT-4V(ision) is a Generalist Web Agent, if Grounded (SeeAct) (arXiv 2401.01614, ICML 2024)
4. Xie, Zhang, Chen et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv 2404.07972, NeurIPS 2024)
5. Zhang, Yu & Yang, Attacking Vision-Language Computer Agents via Pop-ups (arXiv 2411.02391, ACL 2025)
6. OWASP Top 10 for LLM Applications
7. OWASP Agentic Security Initiative
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)