Disclosure Without a Code Path: Reporting Incidents in Model-Driven Systems

When a conventional software system fails in a way others should know about, the path is well trodden. Someone identifies the product and the version, finds the defect in the code, coordinates with the vendor, and publishes an advisory that lets everyone else check whether they are exposed and what to do. Every step of that path assumes the failure has an address: a component, a version, a line. Model-driven systems often fail without one. An agent may have leaked data because of an instruction hidden in a document it read, a tool permission that was broader than intended, a change in the model version underneath it, or a combination that no single owner controls. Run the same task tomorrow and it may not happen again. Yet the people who most need to know, other organisations running the same model, the same tools or the same configuration patterns, are the ones least likely to hear about it. Disclosure for these systems has to describe a behaviour and the conditions that produced it, and it has to do so precisely enough to be useful without becoming a recipe.

Incidents, flaws and vulnerabilities

Three words are often used interchangeably and should not be. An incident is something that happened: a harm, or a near miss, in a real deployment. A flaw is a property of a system that can lead to harm, whether or not it has yet done so; work on coordinated disclosure for AI argues that flaws in machine learning systems extend well beyond security vulnerabilities to include harmful biases, unsafe outputs and other behavioural failures (Cattell et al., 2024). A vulnerability is the narrower class of flaws that an adversary can exploit.

The distinction matters because the regimes that govern each are different. Incidents trigger reporting duties to regulators and affected people. Flaws and vulnerabilities trigger coordinated disclosure to those who can fix them and those who must protect themselves. A single event in a model-driven system can be all three at once: an incident that revealed a flaw, which turned out to be exploitable. A good disclosure process recognises which of the three it is dealing with at each stage, because each audience needs something different.

Why model-driven failures have no code path

Conventional advisories work because the cause of a failure can be located and versioned. In a model-driven system, the behaviour that caused harm usually comes from one or more of four places, and only one of them is code in the ordinary sense.

The model itself, meaning its weights and version, shapes what the system tends to do, and a change of version can change behaviour without any change to the surrounding application. The runtime configuration, meaning the system instructions, the tools the agent is granted and their scopes, retrieval settings, guardrail settings and thresholds, determines what the model is allowed and encouraged to do. The context of the moment, meaning the documents, web pages, messages and tool outputs the model read during the task, can carry instructions that the model follows as if its operator had issued them (Greshake et al., 2023). And the orchestration code that connects these pieces is the only part with a conventional code path.

The practical consequence is that an incident report which names only the application and its version omits most of what an outsider needs to know. The parts that change agent behaviour at runtime, which the operator controls and outsiders cannot see, are exactly the parts that determine whether someone else is exposed.

The reproducibility problem

A software advisory is credible partly because the failure can be reproduced. Model behaviour is often not deterministic: the same inputs can produce different outputs, and a harmful outcome may appear only some of the time. A report that says the agent did something harmful once, with no way to show it again, is easy to dismiss and hard to act on.

Two practices make model-driven incidents reproducible enough to be useful. The first is to capture the full runtime context at the time of the incident: the model version, the configuration, and the exact inputs and tool outputs the model saw, preserved at the action gateway rather than reconstructed later. The second is to report behaviour as a rate rather than an anecdote. Re-run the triggering conditions many times in an isolated environment with harmless stand-ins and report how often the behaviour recurs, with the number of trials. A behaviour that recurs one time in five under stated conditions is something another organisation can test for; a single screenshot is not.

Reproduction also has a limit. The aim is to let others confirm their exposure and validate a mitigation, not to publish a working method for causing the harm. The record proposed later separates the reproducible conditions, which are shared with those who need them, from the payload details, which are shared only on a need-to-know basis.

The reporting regimes, compared

Several regimes now exist, and they differ in who must report, to whom, and what triggers a report. They are complementary rather than alternatives, and a single serious event may fall under more than one.

European Union law now imposes mandatory reporting for some AI systems. Providers of high-risk AI systems must report serious incidents to the relevant market surveillance authorities within deadlines measured in days, and providers of general-purpose AI models with systemic risk must track, document and report relevant information about serious incidents and possible corrective measures to the AI Office without undue delay (Regulation (EU) 2024/1689). The regulation defines a serious incident by its consequences, such as serious harm to health, serious and irreversible disruption of critical infrastructure, or infringement of fundamental-rights obligations.

Voluntary sharing complements mandatory reporting. In the United States, a collaboration playbook from the national cyber defence partnership sets out how industry and government can voluntarily share information about AI-related cybersecurity incidents and vulnerabilities (CISA JCDC, 2025). Public incident collections such as the AI Incident Database, modelled on the incident databases of mature safety industries such as aviation, catalogue real-world AI failures so that the same mistakes are not repeated (McGregor, 2021). And proposals for third-party flaw disclosure for general-purpose AI call for standardised flaw reports, broader safe harbours for good-faith researchers, and disclosure coordination infrastructure across providers (Longpre et al., 2025).

What usually goes missing: the runtime details

Comparing these regimes against the four places a failure can live reveals a consistent gap. Most reports describe the outcome and name the product. Few capture the runtime details that changed the agent's behaviour: which model version was running, what the system instructions said, which tools the agent could call and with what scope, what guardrails were active and at what thresholds, and what external content the agent read just before it acted.

Those details are the difference between a report that helps and one that only informs. Another organisation running the same model with broad tool grants needs to know that the incident depended on tool scope, not on the model alone. A provider needs to know which configuration patterns made a behaviour more likely. A regulator needs to know whether the harm came from the model, the deployment, or the content. Without the runtime details, each of these readers is left to guess.

The same details are often sensitive. System instructions can reveal business logic; tool scopes can reveal architecture; the triggering content may itself be harmful. That is an argument for structured, tiered disclosure, not for leaving them out.

A minimum disclosure record

Drawing these threads together, this article proposes a minimum record for incidents in model-driven systems. It is a synthesis, intended to sit alongside whatever format a regime requires, and it is organised in two tiers: a shareable tier that lets others assess exposure, and a restricted tier shared only with those who must reproduce or fix the problem.

The shareable tier states the outcome and its severity; the affected system type; the model provider and version; the classes of tools involved and their scope, without secrets; the category of the triggering content, such as a retrieved web page or an inbound email; the conditions under which the behaviour recurred and the recurrence rate with the number of trials; the controls that were in place and which of them failed; and the mitigations applied and whether they were validated.

The restricted tier holds the exact configuration, the exact triggering inputs, and the captured trajectory from the gateway's intent log. It is shared with the model provider, affected parties and authorities as required, under the coordination norms that already govern vulnerability disclosure (Cattell et al., 2024; Longpre et al., 2025).

Making disclosure routine inside an organisation

Disclosure is only as good as the evidence captured before anyone knew it would be needed. The most important preparation is architectural: every agent action should pass through a gateway that records the model version, the configuration fingerprint, the inputs and tool outputs the model saw, and the action it attempted. With that record, an incident report can be assembled in hours; without it, the runtime details are lost the moment the session ends.

The second preparation is procedural. Decide in advance who classifies an event as an incident, flaw or vulnerability; which regimes apply to which systems; who holds the authority to report externally; and how the restricted tier is protected. Rehearse it with a tabletop exercise built around a harmless canary incident, so that the first real report is not also the first attempt.

The third is reciprocal. An organisation that wants useful reports from others should contribute useful reports itself, to its model providers, to sector partners and, where appropriate, to public collections. Shared learning only works if the people who see failures write them down in a form others can use.

Limitations and threats to validity

The regulatory summary here is for orientation only. Obligations depend on the exact classification of a system and the precise text of the law, and organisations should rely on the source texts and legal advice rather than this summary.

The proposed disclosure record is a synthesis, not an adopted standard, and it has not been tested across a population of real incidents. Its tiering reflects a judgement about the balance between helping defenders and enabling misuse, which reasonable practitioners may draw differently.

Recurrence rates depend on the test environment. A behaviour that recurs at a given rate in a sandbox with stand-in tools may recur more or less often in production, so rates should be reported together with the conditions that produced them.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

SHAREABLE TIER
  Outcome and severity: 
  Classification: incident / flaw / vulnerability (all that apply)
  System type and role of the agent: 
  Model provider and version: 
  Tool classes involved and their scope (no secrets): 
  Triggering content category (e.g. retrieved web page, inbound email, tool output): 
  Conditions under which it recurred: 
  Recurrence rate and number of trials: 
  Controls in place / which failed: 
  Mitigations applied / validated? (how): 
  Regimes notified (mandatory / voluntary) and dates: 

RESTRICTED TIER (need-to-know)
  Configuration fingerprint and exact configuration: 
  Exact triggering inputs (stored securely, not inline): 
  Gateway intent log of the trajectory: 
  Recipients and coordination timeline: 

Glossary

Incident

An actual harm or near miss that occurred in a real deployment.

Flaw

A property of an AI system that can lead to harm, including but not limited to security vulnerabilities.

Vulnerability

A flaw that an adversary can deliberately exploit.

Coordinated disclosure

A process in which those who find a flaw share it first with those able to fix it, then publish once protections are available.

Runtime configuration

The settings that shape an agent's behaviour while it runs, such as system instructions, tool grants and scopes, retrieval settings and guardrail thresholds.

Recurrence rate

How often a behaviour reappears when its triggering conditions are re-run, reported with the number of trials.

Serious incident

In EU AI law, an incident or malfunction with severe consequences such as serious harm to health, critical-infrastructure disruption or fundamental-rights infringement.

Configuration fingerprint

A stable identifier, such as a hash, of the exact runtime configuration in force when an action occurred.

References

1. Regulation (EU) 2024/1689 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act)
2. CISA Joint Cyber Defense Collaborative, AI Cybersecurity Collaboration Playbook (2025)
3. McGregor, Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database (arXiv 2011.08512, AAAI 2021)
4. Longpre et al., In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI (arXiv 2503.16861, 2025)
5. Cattell et al., Coordinated Flaw Disclosure for AI: Beyond Security Vulnerabilities (arXiv 2402.07039, 2024)
6. Greshake et al., Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (arXiv 2302.12173, 2023)