A Reporting Standard: Datasets, Seeds, Judge, Confidence Intervals, and Threats to Validity

A field advances when its results can be rerun and its claims contested, and it stalls when they cannot. AI red-teaming is currently closer to the second state: attack-success rates are announced constantly, but the reports rarely carry enough to reproduce them — which dataset, drawn how, scored by what judge, with what uncertainty, under what settings. The remedy is not more experiments but better reports. This article proposes a reporting standard: a fixed, checkable set of fields that any red-team result should carry so that a competent reader can reconstruct the measurement and challenge it. Reproducibility, on this view, is a property of the document, and the standard is the minimum a document must contain to have it.

Reproducibility is a property of the report

An experiment that cannot be rerun from its report is, for scientific purposes, a rumor. The information that makes a result reproducible does not live in the run; it lives in what the run wrote down. Two red-team studies can perform identical work and differ entirely in credibility because one recorded its dataset, seeds, judge, and intervals and the other reported a bare percentage. The gap between them is a documentation gap, and it is the one this standard closes.

Holistic evaluation practice already models the discipline for language-model benchmarking: fix the scenarios, publish the metric definitions, and report across conditions so that a result is a reconstructable object rather than a headline. The NIST adversarial-machine-learning taxonomy supplies the other half — a shared vocabulary for the adversary's goal, capability, and knowledge — so that two reports describe the same threat in the same terms. A reporting standard for attacks is the intersection: the terms plus the artifacts.

The standard below is deliberately minimal and enforceable. Each field exists because omitting it has been shown, elsewhere in this measurement literature, to make a rate uninterpretable or irreproducible. Together they form a checklist a reviewer could apply mechanically.

The five required components

A reproducible red-team report rests on five layers, each addressing a distinct way a rate goes wrong. Dataset provenance fixes what was measured and the population it represents. Seeds and settings fix the stochastic conditions so the run can be repeated. The calibrated judge fixes how success was decided and how much to trust that decision. The estimate-with-interval fixes the value and its uncertainty over the correct unit. And the threats-to-validity section fixes the honest boundaries of the claim.

These layers are ordered from the raw material up to the interpretation: data at the base, then the conditions applied to it, then the instrument that labels it, then the statistic that summarizes the labels, then the caveats that bound the statistic. Removing any layer undermines the ones above it — a perfect interval computed over an undocumented dataset is precise nonsense, and a well-provenanced dataset scored by an uncharacterized judge is uninterpretable no matter how careful the statistics.

The remainder of this piece specifies each layer to the level of detail a reviewer needs, then assembles them into a template and a gate.

Dataset provenance and seeds

The dataset layer answers what was attacked and where the trials came from. A report should carry a dataset description precise enough to redraw the sample: the source of the prompts, the sampling frame, the size, the unit of analysis, and a contamination note stating whether the material could plausibly appear in the target's training data. Without this, the population the rate estimates is unknown, and the rate generalizes to nothing in particular.

The seeds layer answers under what stochastic conditions. Language-model outputs depend on decoding settings — temperature, sampling method, and any random seed — and on the model version and date. A rate produced at high temperature is a different measurement from one at low temperature, and neither is reproducible unless the settings are recorded. The report should fix the decoding configuration, the seeds, the model identifier and version, and the date of the run, so that the stochastic surface is pinned down.

Together these two layers make the experiment re-runnable in principle: a reader with the same dataset description, the same settings, and the same target could reconstruct the trials. That is the precondition for everything above them.

The judge and the estimate, disclosed

The judge layer answers how success was decided. A report should publish the success rubric, state whether the grader was human or automated, and — critically — report the grader's agreement against a human reference, such as a chance-corrected kappa, along with its confusion-matrix operating characteristics. A rate whose judge is undisclosed cannot be reconciled with any other rate, because the two may be measuring success differently; a rate whose judge is characterized can be corrected and compared.

The estimate layer answers what the number is and how much to trust it. A report should state the estimand — the unit of analysis and the population — and give the point estimate with its denominator and a coverage-valid interval such as Wilson's, computed over the correct independent unit and widened for clustering where paraphrase families are pooled. The interval, not the point, carries the claim, and its method must match the data.

These two layers are where most published rates fail: the judge is unnamed and the interval is absent. Supplying both is the difference between a number that can be trusted and one that must be taken on faith.

The reproducibility pipeline

The layers assemble into a pipeline that a reader can traverse to rerun the study. Start from the dataset description and redraw the sample; apply the recorded seeds and settings to regenerate the trials; score them with the disclosed judge; and recompute the estimate and interval over the stated unit. If each stage is documented, the reader arrives at the same number within sampling noise; if any stage is missing, the traversal breaks and the result is unreproducible at that point.

This pipeline doubles as a diagnostic. When two studies disagree, walking the pipeline localizes the disagreement: different datasets, different settings, different judges, or different statistics. Without the documented stages there is nothing to walk, and the disagreement is irresolvable. The standard, in other words, does not just enable reproduction — it enables debugging the science.

The cost of maintaining the pipeline is a modest reporting overhead. The return is that every number becomes a reconstructable, contestable object rather than a claim to be believed or dismissed.

A reviewer's gate

The standard becomes useful when it is enforced, and enforcement is a gate a reviewer applies before a rate is accepted. The gate contrasts two documents: an anecdote — a bare percentage with no estimand, no judge, no interval — and a standard report that carries all five layers. The reviewer's job is binary and mechanical: does the report contain each required field, or not. A missing field is not a stylistic lapse; it is a hole through which reproducibility escapes.

This mechanical enforceability is the point of a standard. It removes the burden of judgment about whether a result is trustworthy and replaces it with a checklist about whether the result is documented. A well-documented result may still be wrong — but it can be checked, which is all science asks. A poorly documented result cannot be checked and therefore cannot be trusted regardless of the number.

The gate is cheap to run and expensive to fail, which is exactly the incentive a healthy measurement culture needs. Publish the five layers, or the rate does not pass.

Threats to validity, and using the standard

The final layer is the report's own honesty about its limits. A threats-to-validity section should state the known weaknesses: the narrowness of the population, the residual bias of the judge, the assumptions behind the interval, and the axes of generalization that were not tested. Naming these does not weaken a result; it bounds it correctly, and a bounded result is more useful than an overclaimed one. A report without a limitations section is implicitly claiming there are no limitations, which is never true.

Using the standard is straightforward: treat the five layers as required fields, fill each with the specific artifact it names, and run the reviewer's gate before publishing or comparing any rate. The overhead is a page of documentation; the payoff is a number other teams can reproduce, reconcile, and build on. Adopted broadly, the standard turns a pile of incomparable percentages into a cumulative body of evidence.

The standard applies unchanged to any binary AI-security outcome. Whether the claim is about jailbreaks, extraction, poisoning, or evasion, the same five layers make it reproducible, and the same gate keeps the field honest. Report the estimand, the conditions, the instrument, the interval, and the limits — or do not report a measurement at all.

1. Provide dataset provenance detailed enough to redraw the sample, with a contamination note.
2. Record seeds, decoding settings, model version, and run date.
3. Disclose the judge rubric and its agreement (kappa) and confusion matrix.
4. Report the estimand, point estimate, denominator, and a coverage-valid interval.
5. Include an explicit threats-to-validity section.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

RESULT: <attack name>
1. DATASET
   source: <where prompts came from>
   sampling frame: <how any trial could be drawn>
   size / unit: <n> / <per-prompt | per-family | per-target | per-conversation>
   contamination: <could this appear in target training data?>
2. CONDITIONS
   model + version + date: <id / vX / YYYY-MM-DD>
   decoding: <temperature, method, seed(s)>
3. JUDGE
   rubric: <one testable success sentence>
   type: <human | automated>
   agreement: kappa=<value> ; confusion: TPR=<..> FPR=<..>
4. ESTIMATE
   estimand: <unit over population>
   value: p_hat=<..> ; n=<..> ; CI(95%, Wilson)=[<lo>, <hi>]
5. THREATS TO VALIDITY
   - <population narrowness>
   - <residual judge bias>
   - <interval assumptions / clustering>
   - <generalization axes not tested>

Glossary

Reporting standard

A fixed set of fields a result must carry so another team can reproduce and contest it.

Dataset provenance

The documented source, sampling frame, size, unit, and contamination status of the trials.

Decoding settings

The temperature, sampling method, and seeds that fix a language model's stochastic output.

Calibrated judge

A grader whose rubric, agreement (kappa), and confusion matrix against a human reference are reported.

Estimand

The population quantity being estimated, fixed by the unit of analysis and the population.

Threats to validity

An explicit statement of a result's known weaknesses and the boundaries of its claim.

Reviewer's gate

A mechanical check that each required reporting field is present before a rate is accepted.

References

1. Liang et al., Holistic Evaluation of Language Models / HELM (arXiv 2211.09110)
2. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology
3. Wilson, Probable Inference, the Law of Succession, and Statistical Inference (JASA 1927)
4. Cohen, A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement 1960)
5. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)