UI-Grounding Attacks: Decoy Elements, Overlay and Z-Order Tricks, and Pixel-Level Steering

A screen-driven agent can plan a task flawlessly and still do the wrong thing, because between the plan and the world sits a fragile translation: grounding, the step that turns 'click the submit button' into an actual click at an actual pixel. The capability literature is blunt about how fragile this is. In the SeeAct study, a strong multimodal model planned web tasks well yet succeeded far more often only when a human manually grounded its plans into on-screen actions, and OSWorld found grounding to be a primary reason agents failed real operating-system tasks that humans completed easily. An attacker reads those results as an invitation. To subvert such an agent you do not need to corrupt its reasoning, jailbreak its instructions, or change its goal; you only need to move where its correct goal lands. This article is about that move — the family of UI-grounding attacks that leave the plan pristine and bend the mapping from intent to element, so a faithful agent executes a hostile action while every trace of its thinking looks exactly right.

Grounding Is the Weak Joint

Grounding is the transition in a computer-use agent's loop that converts an abstract intended target — a described button, a field, a menu item — into a concrete, actionable location, whether an element in a structured tree or a set of pixel coordinates. It is the joint between deciding and acting, and it is where an intention becomes a physical effect on real software. Naming it precisely matters because the attacks in this article all operate on this one transition and nowhere else.

The empirical record makes grounding the natural target. Zheng, Gou, Kil and colleagues showed in their SeeAct work that a capable model's planning could be strong while its ability to turn a plan into the right on-screen action lagged badly, with grounding the dominant error source; OSWorld reported the same pattern at operating-system scale, where the best evaluated agent solved only a small fraction of tasks humans found routine, again bottlenecked on grounding. These are non-adversarial measurements, but they establish that even without an attacker, agents frequently ground to the wrong place.

That baseline unreliability is the attacker's cover. A defender trying to distinguish a malicious mis-ground from an ordinary one is looking for a signal inside a channel that is already noisy by design. The rest of this article develops the specific ways an adversary bends grounding, and each should be read against this backdrop: the attack does not have to be reliable to be dangerous, because it hides in error the system already tolerates.

The Grounding Channels

Agents ground through up to three channels, and each is a distinct attack surface. The visual channel grounds from raw pixels: the model looks at a rendered screenshot and locates the target by appearance and position. The structural channel grounds from a machine-readable representation — an accessibility tree or a document object model — locating the target by its role, name, and hierarchy. The semantic channel grounds from text: the model matches its intended target's description against labels and captions on screen. Real systems often fuse these, and SeeAct found that combining structure and visuals grounded better than either alone.

Fusion helps capability but multiplies the attack surface, because an adversary can target whichever channel the agent trusts most or whichever is easiest to manipulate in a given context. Pixels can be perturbed; trees can be mislabeled; text can be spoofed. Worse, the channels can be made to disagree deliberately, so that the element the agent sees, the node it reads, and the label it matches are three different things — and the agent resolves the conflict in the attacker's favor.

Treating the channels separately is what makes a defense tractable. A control that only validates pixels does nothing against tree poisoning; a control that only checks labels is blind to an overlay. The taxonomy that follows is organized so that each attack is tied to the channel it abuses, which is the first step toward covering all three.

Decoy Elements: Grounding to the Lookalike

The simplest grounding attack places a decoy: an element that resembles the agent's intended target closely enough that grounding selects it instead of the genuine one. If the agent intends to click a control described as 'submit', an attacker who renders a second, similar-looking or similarly-labeled control near the real one creates an ambiguity the grounding step must resolve, and the decoy is designed to win that resolution. The plan said submit; the decoy answered.

Decoys exploit the fact that grounding is a similarity judgment, not an authenticated lookup. The agent has no registry of legitimate controls to check against; it picks the element that best matches its description of the target, and 'best match' can be engineered. Because the decoy is a real, rendered element, nothing about the agent's perception is malformed — it perceives two plausible targets and chooses one, exactly as it would in a benign but cluttered interface.

The danger scales with the consequence of the decoyed action. A decoy that captures a click destined for a harmless control is a nuisance; a decoy that captures a click destined for a confirm, send, or authorize control converts a correct plan into a harmful act. This is why decoy resistance cannot be uniform — it must be strongest exactly where the intended action is most consequential.

Overlay and Z-Order Tricks: Seeing One Thing, Clicking Another

A second family separates what the agent perceives from what its action actually reaches. In any layered interface, elements stack in a front-to-back order, and the element that visually dominates a region need not be the one that receives a click at that region. An attacker can exploit this the way clickjacking exploits human users: render a benign, salient element for perception while a different element occupies the actual hit target, or place a transparent layer that intercepts the action. The agent grounds to what it sees; the interface delivers the action elsewhere.

This attacks the seam between grounding and acting. Grounding may correctly identify the visible target, but the coordinate it produces is redeemed by the environment's own layering rules, which the attacker controls. The agent's perception and the world's response are made to disagree, and the disagreement is invisible in the screenshot the agent reasoned over, because the malicious layer is either transparent or below the salient one.

The observe stage compounds the problem. After the action, the agent reads a new screen the same environment renders, so the feedback can be arranged to look as if the intended element was activated. The loop closes on a lie: perceived the right target, clicked the wrong one, and observed a success.

Pixel-Level Steering of the Grounding Model

Where grounding relies on the visual channel, the grounding model is itself a target. Vision models are known to be sensitive to carefully constructed inputs, and a screen is an input the environment fully authors. An adversary can, in principle, craft on-screen content whose appearance nudges a vision-based grounding model toward an attacker-chosen element — steering the model's localization without changing anything a casual human observer would notice. This is the interface analog of an adversarial example: the picture looks ordinary, but it is optimized to move the model's output.

There is direct empirical support that deliberately designed on-screen content can reliably divert computer-use agents. Zhang, Yu and Yang demonstrated that adversarial elements added to agent testing environments caused vision-language agents to click them at a high rate and sharply reduced task success, and that naive instructions to ignore such content did not defend against it. While their specific artifacts are their own study, the finding generalizes to this section's point: the grounding model's visual judgment can be steered by environment-authored pixels, and telling the agent to be careful is not a control.

The severity here is that the attack targets the model, not just the layout. A decoy or overlay can be caught by validating elements structurally, but a perturbation that steers the grounding model may leave the structural representation untouched while corrupting the visual localization. Defenses must therefore not depend on the visual channel alone.

Accessibility-Tree and Semantic Poisoning

The structural channel is not automatically trustworthy just because it is machine-readable. An accessibility tree or document representation is populated by the same environment the agent does not control, so its roles, names, and labels can be authored adversarially. An attacker can give a hostile element the role and name of the control the agent seeks, so a grounding step that trusts the structure resolves to the wrong node — a structural decoy rather than a visual one.

Semantic poisoning is the text-channel version: labels and captions crafted to match the agent's description of its target. Because grounding often matches the agent's intended-target phrase against on-screen text, an element labeled to echo that phrase can capture the grounding even when its appearance and position are unremarkable. The agent's own task description becomes the selector the attacker exploits.

These attacks matter most for agents that lean on structure or text to compensate for weak visual grounding — precisely the fusion that improves capability. The defensive implication is that no single channel can be treated as ground truth: structure, pixels, and labels must corroborate one another, because each in isolation is authored by the untrusted environment.

Why Grounding Attacks Are Stealthy

Grounding attacks are unusually hard to detect because they hide in two blind spots at once. The first is the natural error distribution: agents mis-ground often even without an adversary, so a maliciously mis-grounded action does not stand out against a baseline of frequent honest mistakes. There is no clean 'correct behavior' signature to deviate from. The second is the reasoning trace: because the plan is untouched, any inspection of the agent's stated reasoning shows a correct intention. The corruption lives entirely in the final intent-to-coordinate step, which the reasoning does not narrate.

This defeats two defenses practitioners often reach for. Watching the agent's chain of thought will not reveal a grounding attack, because the thought was right; only the landing was wrong. And anomaly detection on outcomes is weak when wrong outcomes are already common. The attack is, in effect, camouflaged by the very unreliability that the capability benchmarks documented.

The lesson is that grounding must be defended at the grounding step, not upstream in reasoning or downstream in coarse outcome monitoring. The controls that work are the ones that constrain, corroborate, and confirm the intent-to-element mapping itself — because that is the only place the attack actually lives.

Defenses for the Ground Stage

A coherent defense treats grounding as a security-critical resolution and hardens it directly. Cross-channel agreement is the backbone: require that visual, structural, and semantic grounding corroborate before a consequential action, so an attack that bends one channel is caught by disagreement with the others. Expected-target constraints narrow the candidate set to elements consistent with the current task, turning an injected decoy or poisoned node into an out-of-set anomaly rather than a valid choice.

Confirmation and reversibility gate the consequences. High-consequence controls should not be reachable on a single grounded click of ambiguous confidence; a second check, a human confirmation, or a reversible action path denies a bent grounding its final effect. Robustness testing closes the loop before deployment: adversarial evaluation of the grounding model, treating high divertibility as a release blocker, ensures the visual channel is not trivially steerable in the first place.

Finally, low grounding confidence on a sensitive target should be a stop condition, not a guess. An agent that proceeds into its own error distribution when the stakes are high is doing the attacker's work; an agent that halts and escalates when confidence is low on a consequential control denies the attack the one thing it needs — an executed action.

What This Means for the AI-Agent Stack

Grounding is the agent's hands, and a stack that pours its security budget into the model's reasoning while leaving grounding unguarded has protected the wrong stage. The OWASP Agentic guidance's emphasis on excessive agency lands precisely here: an agent that can click anything, grounding through channels the environment authors, has an agency problem that no amount of well-behaved reasoning fixes. The intent-to-element step is where agency becomes action, so it is where agency must be constrained.

The portable takeaway for any screen-driven agent is that a correct plan is not a safe plan. Assume the environment can bend where a plan lands; make the grounding channels corroborate; restrict targets to what the task expects; and confirm before consequential actions are realized. These controls do not make grounding perfect — the capability benchmarks show perfection is far off — but they deny an attacker the ability to convert the agent's honest reasoning into a hostile act through the one fragile joint between intent and click.

The single idea to carry is that grounding attacks are invisible to trace inspection and camouflaged by natural error, so they must be met where they live: at the mapping from intent to element, corroborated across channels and confirmed before it lands.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE cross-channel grounding agreement.
v = ground_visual(screen, intent)     # pixel-based candidate
s = ground_structural(tree, intent)   # tree-based candidate
t = ground_semantic(labels, intent)   # label-based candidate
if is_consequential(intent):
    if not (v == s == t):
        halt('channel disagreement')  # possible decoy/overlay/poison
    if confidence(v, s, t) < THRESH:
        require_confirmation()
act_on(agree(v, s, t))

task: submit-support-ticket
expected_targets: [ticket-form.submit, ticket-form.field.*]
reject_if: target NOT IN expected_targets   # injected decoy => out of set
consequential: [ticket-form.submit]
consequential_requires: cross_channel_agreement + confirmation
on_low_confidence: halt_and_escalate

Glossary

Grounding

The stage that converts an intended target into a concrete on-screen element or pixel coordinate.

Grounding channel

A source an agent grounds from — visual pixels, a structured accessibility or document tree, or semantic labels.

Decoy element

An engineered lookalike control placed so the agent's grounding selects it instead of the genuine target.

Z-order / overlay attack

Manipulating interface layering so the element perceived as the target differs from the one that receives the action.

Pixel-level steering

Adversarial on-screen content optimized to move a vision-based grounding model's localization toward an attacker-chosen element.

Tree / semantic poisoning

Authoring adversarial roles, names, or labels so structural or text grounding resolves to a hostile element.

References

1. Zheng, Gou, Kil, Sun & Su, GPT-4V(ision) is a Generalist Web Agent, if Grounded (SeeAct) (arXiv 2401.01614, ICML 2024)
2. Xie, Zhang, Chen et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv 2404.07972, NeurIPS 2024)
3. Zhang, Yu & Yang, Attacking Vision-Language Computer Agents via Pop-ups (arXiv 2411.02391, ACL 2025)
4. Zhou, Xu, Zhu et al., WebArena: A Realistic Web Environment for Building Autonomous Agents (arXiv 2307.13854, ICLR 2024)
5. Greshake, Abdelnabi, Mishra et al., Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2302.12173, 2023)
6. OWASP Top 10 for LLM Applications
7. OWASP Agentic Security Initiative
8. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)