Instructions Hidden in Images: Perturbation and Steganographic Injection Robust to Resizing

The unsettling thing about an image payload is that you can look straight at it and see nothing wrong. A photograph of a coffee cup, a screenshot of a dashboard, a scanned receipt — each can carry an instruction that a vision-language model will read and act on while a human sees only the picture. This is not magic; it is a direct consequence of how these models work. They convert an image into a sequence of embeddings that share representational space with text, and any signal in the pixels that pushes those embeddings toward instruction-like meaning becomes, effectively, text the model reads. The attacker's craft is not in fooling a human — that part is easy — but in making the hidden instruction survive everything the system does to an image before the model ever sees it.

Two ways to hide an instruction in pixels

There are two broad families of image injection, and they differ in whether the payload is perceptible. The first is overt-but-exploited text: the instruction is actually visible in the image — small print in a corner, faint low-contrast text, words embedded in a diagram — and the attacker relies on the model reading it even when a rushed human would not. This works because vision-language models are trained to read text in images, so legible-to-the-model content the operator never intended as instructions is dutifully ingested.

The second family is imperceptible perturbation: the attacker adds a carefully computed, low-amplitude noise pattern to an otherwise innocent image so that the model's encoding shifts toward a target instruction, while to a human the image looks unchanged. Bagdasaryan and colleagues demonstrated this cross-modal indirect injection directly — an adversarial image (or sound) that causes a multimodal model to follow an embedded instruction — establishing that perception itself is the vulnerability rather than any single rendering trick.

Both families exploit the same downstream fact established by the cross-modal threat model: whatever the image resolves to enters the model's context as trusted-looking content. The difference is operational. Visible-text payloads are trivial to create and trivial to detect if anyone looks; perturbation payloads are hard to detect but fragile, because the tiny signal that carries them is exactly what lossy preprocessing destroys.

The real constraint: surviving the preprocessing pipeline

Every production image passes through transformations before it reaches the model: it is decoded, resized to the model's fixed input resolution, often re-compressed as JPEG, sometimes cropped or padded, and normalized. Each of these is a lossy operation that discards high-frequency detail — and high-frequency detail is precisely where an imperceptible perturbation lives. This is why robustness, not stealth, is the attacker's binding constraint: a payload that steers the model at full resolution but dissolves under a bilinear downscale is useless in practice.

The attacker therefore optimizes not against the model alone but against the composition of the preprocessing pipeline and the model. Conceptually, they search for a perturbation that maximizes the chance the model emits the target behavior after the image has been degraded by the expected transforms. This is the same expectation-over-transformations idea that makes physical-world adversarial examples robust, applied here to the digital preprocessing path.

The defensive leverage in this constraint is significant. Because the attack must survive a specific pipeline, a defender who randomizes or strengthens that pipeline — introducing transforms the attacker did not optimize against — can break payloads without ever detecting them. Robustness cuts both ways: the attacker needs it, and the defender can deny it.

The perceptibility-robustness trade-off

An attacker crafting an image payload faces a fundamental tension. Making a perturbation more robust to lossy transforms generally means increasing its amplitude or concentrating it in lower-frequency components — which makes it more perceptible to a human and more detectable by a defender. Making it less perceptible means smaller, higher-frequency signals that are more easily destroyed by downscaling and compression. There is no free lunch; every image payload sits somewhere on this trade-off.

This structure is useful defensively because it tells you where to spend effort. Against the stealthy end (small, high-frequency, fragile), normalization alone is often sufficient. Against the robust end (large, low-frequency, perceptible), perceptual anomaly checks and human-in-the-loop review for high-impact actions catch what normalization might pass. Framing the defense as covering both ends of the trade-off — rather than chasing a single detector — is what makes it durable.

The matrix below places the two payload families and their operating points, and identifies the control that dominates each quadrant.

The exchange: how an image payload actually fires

It helps to trace the full exchange. The attacker places an adversarial image where the target agent will ingest it — a shared document, a web page the agent browses, an uploaded file. The agent, executing an ordinary task (describe this, extract the total, summarize the page), passes the image through preprocessing to the vision-language model. If the payload survived, the model's encoding surfaces the embedded instruction into the context, and the agent — unable to distinguish this from a legitimate instruction — acts on it, potentially invoking a tool or exfiltrating data.

The critical observation is that no step in this exchange is anomalous from the system's point of view. The agent was asked to look at an image and it looked at an image; the tool call it makes afterward is a normal tool call. This is why the defense cannot rest on catching a suspicious event — there is no suspicious event — and must instead deny the channel and contain the outcome. Structural trust-labeling of perceived content and least-privilege tool access are the controls that turn a successful firing into a non-event.

The sequence below shows the exchange and marks the single point where the defense must intervene: at the boundary where perceived content becomes context.

Why detection alone fails, and what replaces it

It is tempting to answer image injection with a classifier that flags malicious images. This fails for a structural reason: the space of imperceptible perturbations is effectively unbounded, an attacker can adapt to any fixed detector, and visible-text payloads can be arbitrarily rephrased. A detector raises the attacker's cost but cannot be the primary control, and treating it as such produces a brittle defense that fails silently the first time an attacker adapts.

The durable replacement is a layered stance. First, deny the channel: normalize and randomize preprocessing so fragile payloads dissolve, and OCR-plus-label so visible-text payloads are surfaced as data, not obeyed. Second, separate trust: mark all perceived content as untrusted so an instruction-shaped span from an image never competes as a command. Third, contain the blast radius: least privilege, confirmation for high-impact actions, and egress controls so a payload that does fire cannot reach anything valuable. Detection sits on top of these as a cost-raiser, not underneath them as a foundation.

This is the same conclusion the broader injection literature reaches, and OWASP's placement of prompt injection at the top of the LLM risk list reflects that no input filter has proven sufficient on its own. The image channel is one more delivery mechanism for a problem whose solution is architectural.

Agent implications and a hardening posture

For an autonomous agent that browses, opens files, or processes user uploads, image injection is a live path to unauthorized action. The posture that neutralizes it is consistent with the whole multimodal defense: perceive through a single hardened path, label everything perceived as untrusted, minimize authority, and confirm consequential actions. An agent built this way can be fooled by an image and still cause no harm, which is the only reliable success criterion given that perfect input hygiene is unattainable.

The synthesis is that image payloads are governed by robustness, and robustness is a shared battleground. The attacker must invest in surviving the pipeline; the defender wins by making the pipeline hostile to fragile payloads and the agent's authority too small for robust ones to exploit. Neither detection nor stealth is the decisive factor — the decisive factor is who controls the transform chain and the tool permissions, and that is the defender.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE PROBE — does our pipeline change model output under transforms?
canary_images = load_benign_images()   # no payloads, just test inputs

function robustness_probe(model, pipeline, images):
    for img in images:
        base = model(pipeline.normalize(img))
        for t in [downscale, jpeg_recompress, jitter_crop]:
            variant = model(pipeline.normalize(t(img)))
            if diverges(base, variant):
                flag(img, t)   # transform meaningfully changes output
    # Strong divergence under cheap transforms == pipeline is hostile
    # to fragile perturbations (good). Use to tune normalization.

image_policy:
  normalization:
    downscale_to_model_resolution: true
    reencode: jpeg
    quality_jitter: [70, 90]        # randomize compression
    rerasterize_untrusted: true
  ocr:
    extract_text: true
    label: untrusted_data           # never instructions
  framing:
    delimit_perceived: required
    concatenate_raw: forbidden
  authority:
    post_image_tool_calls: least_privilege
    high_impact: require_confirmation

Glossary

Vision-language model

A model that encodes images into the same representational space as text, letting image content act like readable tokens.

Adversarial perturbation

A small, deliberately computed change to an image's pixels that shifts the model's output while remaining imperceptible to humans.

Steganography

Hiding information within a carrier (here an image) so its presence is not apparent to an observer.

Preprocessing pipeline

The fixed sequence of transforms — decode, resize, compress, crop, normalize — applied to an image before the model encodes it.

Expectation over transformations

Optimizing a payload to remain effective on average across a distribution of transforms, making it robust to preprocessing.

Perceptibility-robustness trade-off

The tension whereby making an image payload survive lossy transforms tends to make it more visible and detectable.

Channel denial

Defending by normalizing and randomizing the input pipeline so payloads dissolve, rather than by detecting them.

Blast radius

The harm reachable after a successful injection, bounded by the agent's tools, credentials, and permitted actions.

References

1. Bagdasaryan et al., Abusing Images and Sounds for Indirect Instruction Injection (arXiv 2307.10490)
2. Carlini & Wagner, Audio Adversarial Examples: Targeted Attacks on Speech-to-Text (arXiv 1801.01944)
3. Greshake et al., Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2302.12173)
4. OWASP Top 10 for LLM Applications (LLM01 Prompt Injection)
5. MITRE ATLAS (Adversarial Threat Landscape for AI Systems)