Sampling and Confidence: Wilson Intervals, Per-Family Estimation, and Sequential Testing

Suppose a red-team run reports that an attack succeeds on 3 of 20 trials. The point estimate is 15 percent, but that number alone is nearly content-free: it could be consistent with a true rate anywhere from a few percent to well over thirty. The missing quantity is the sampling uncertainty — how much the estimate would wobble if the same experiment were repeated — and it is carried by a confidence interval, not by the point. This article treats an attack-success-rate as what it is, a proportion estimated from a finite sample of Bernoulli trials, and builds the machinery to report it responsibly: an interval method that behaves near zero and one, a correction for the correlated trials that paraphrase families introduce, and a discipline for looking at the data as a campaign accumulates.

The sampling model, stated plainly

Fix the estimand — a population success probability under a declared unit, population, and judge — and call it p. A study draws n trials and counts x successes. If the trials are independent and identically distributed, x follows a binomial distribution with parameters n and p, and the natural estimator is the sample proportion. Its standard error shrinks with the square root of the sample size, which is the precise sense in which more trials buy more precision.

Two features of this model drive everything that follows. First, the variance of the estimator depends on p itself, and is largest near one-half and smallest near zero and one — so the precision of a rate is not uniform across its range. Second, the independence assumption is load-bearing: the standard error formula is only correct if the trials really are independent draws, which, as we will see, red-team samples frequently are not.

The point estimate is therefore just the center of a distribution of plausible values. Reporting it without its spread is like reporting a measurement without its error bar — technically a number, but not yet evidence.

Why the textbook interval fails for attack rates

The interval most people reach for is the Wald interval: the point estimate plus or minus a multiple of its estimated standard error, using a normal approximation. It is taught first and fails exactly where red-teaming operates. Near zero or one — where dangerous-capability attack rates often sit — the symmetric Wald interval spills outside the valid range, producing nonsensical bounds below zero or above one. For small samples it undercovers badly, meaning a nominal 95 percent interval contains the true value far less than 95 percent of the time.

The failure is not a rounding issue; it is structural. The Wald interval assumes the sampling distribution is approximately normal and centered on the estimate, an approximation that collapses when the estimate is near a boundary or the sample is small. A red-team result of 0 successes in 30 trials yields a Wald interval of exactly zero width — the absurd claim that the true rate is known to be exactly zero — which is obviously wrong and dangerously reassuring.

Because the extreme-rate, small-sample regime is the normal habitat of attack measurement, the Wald interval is the wrong default here. The fix is an interval derived without the boundary-breaking approximation.

The Wilson score interval

Wilson's 1927 score interval solves the problem by inverting the score test rather than approximating the estimate as normal. Instead of centering the interval on the point estimate, it finds the set of population rates under which the observed count is not surprising, which keeps the interval inside the valid zero-to-one range and preserves its stated coverage even for small n and extreme rates. It is the well-behaved default that modern statistical guidance recommends over Wald for proportions.

The interval has a closed form: a center that is the observed rate pulled gently toward one-half, and a half-width that accounts for both sampling variance and the pull. For the 0-of-30 case it yields a sensible upper bound well above zero — the honest statement that few or no observed successes are consistent with a small but nonzero true rate. That single property, giving a usable upper bound when nothing was observed, is why it is indispensable for safety-relevant rates.

On a probability scale, the report becomes a point with an asymmetric band around it: the estimate, and the Wilson bounds that say where the true rate plausibly lies. The asymmetry near the boundaries is a feature, not a defect — it reflects the genuine shape of the uncertainty.

Correlated trials: paraphrase families break independence

The independence assumption underlying every interval so far is routinely violated in red-teaming. Attacks are generated in families — a seed prompt and its paraphrases, or a template instantiated many times — and trials within a family are correlated: if one paraphrase succeeds, its siblings are more likely to succeed too. Treating m correlated paraphrases as m independent trials overstates the effective sample size and produces an interval that is too narrow, advertising more precision than the data support.

The magnitude of the distortion is captured by the design effect, which scales the naive variance by a factor that grows with the cluster size and the intraclass correlation — the fraction of variance attributable to the family rather than the individual trial. When correlation is high, a study of a thousand paraphrases from ten seeds carries closer to the information of ten independent trials than of a thousand, and its honest interval is correspondingly wider.

The sources of uncertainty therefore extend beyond raw sample size. Sample size, the base rate through its effect on variance, and the clustering structure all shape how much to trust a rate — and ignoring the last of these is the most common way red-team intervals are quietly too narrow.

Per-family estimation done right

If the estimand is per-family — the probability that a budgeted attacker with a set of paraphrases succeeds — then the family, not the paraphrase, is the unit, and the interval is computed over families. This is the clean case: each family contributes one Bernoulli outcome (did any paraphrase succeed), the outcomes across families are independent, and a Wilson interval over the number of families applies directly. The paraphrases inside a family are budget, not additional evidence about the population.

If the estimand is per-prompt but the sample is clustered, the naive interval must be widened by the design effect, or the estimation must use a cluster-aware method that treats families as the resampling unit. A simple and robust option is to resample whole families rather than individual prompts when building the interval, which automatically respects the correlation structure. The wrong move — the common one — is to pool all paraphrases as if independent and report the resulting too-narrow interval.

The discipline reduces to a single question asked before any interval is computed: what is the independent unit? Answer it honestly, make that unit the atom of the interval, and the correlated-trials trap closes.

Sequential testing: the cost of looking

Red-team campaigns are rarely one-shot; they accumulate trials and the operator watches the rate evolve, often stopping when it crosses a threshold. This repeated inspection has a statistical cost. Each time you test the same hypothesis on accumulating data and allow yourself to stop, you get another chance to cross a significance boundary by luck, so the true false-positive rate rises well above the nominal level. An interval recomputed after every trial and reported the first time it excludes a target value is not a valid interval.

The remedy is to plan the looking. Either fix the sample size in advance and report once, or use a sequential design that spends its error budget across the planned interim analyses so the overall false-positive rate stays controlled. The essential decision at each interim point is disciplined: continue collecting, or stop — and if stopping, do so against a boundary that was set before the data were seen, not against the tempting number on the screen.

This is not an argument against monitoring a campaign; it is an argument for pre-committing to how monitoring may influence conclusions. Exploratory watching is fine; a headline rate harvested from the luckiest peek is not.

Threats to validity, and what to report

The dominant threats here are statistical. A Wald interval on an extreme rate or a small sample gives invalid or zero-width bounds and should never appear in a safety-relevant report. A naive interval on clustered paraphrases understates uncertainty by ignoring the design effect. And an interval harvested from repeated peeking overstates confidence because the looking was not planned. Each of these produces a number that looks precise and is not.

The reportable package is small and strict: the estimand and its independent unit; the point estimate with its denominator; a Wilson (or better) interval computed over the correct unit, widened for clustering when the estimand is per-prompt; and, for campaigns, a pre-registered stopping rule. A rate accompanied by that package states not just a value but how much to trust it, which is the whole purpose of measurement.

The framing generalizes to any binary AI-security proportion — extraction success, trigger activation, evasion — all of which are estimated from finite, often clustered, sometimes sequentially collected samples. The statistics do not care what the attack is; they care how the trials were drawn.

1. Never report a Wald interval for an extreme rate or small sample — use Wilson.
2. Identify the independent unit and compute the interval over it.
3. Widen for clustering (design effect) whenever paraphrase families are pooled.
4. Pre-register the stopping rule for any campaign that watches the rate accumulate.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE / MOCK ONLY — no real target, no attack content
function rate_with_interval(trials, unit):
    if unit == PER_FAMILY:
        outcomes = [any(success(p) for p in fam) for fam in group_families(trials)]
    elif unit == PER_PROMPT_CLUSTERED:
        # resample whole families to respect correlation
        return cluster_bootstrap_wilson(trials, group=family_of)
    else:
        outcomes = [success(t) for t in trials]
    x, n = sum(outcomes), len(outcomes)
    lo, hi = wilson(x, n, z=1.96)      # valid near 0/1 and small n
    return Report(estimate=x/n, n=n, ci=(lo, hi), unit=unit)

Glossary

Sample proportion

The number of successes divided by the number of trials, the natural estimator of a success probability.

Wald interval

The normal-approximation confidence interval for a proportion, which misbehaves for small samples and extreme rates.

Wilson score interval

A confidence interval for a proportion derived by inverting the score test, valid within [0,1] and for small samples.

Coverage

The long-run fraction of computed intervals that contain the true value; a valid 95% interval covers 95% of the time.

Design effect

The factor by which clustering inflates the variance of an estimate relative to simple random sampling.

Intraclass correlation

The fraction of total variance attributable to differences between clusters (e.g. paraphrase families) rather than within them.

Sequential testing

Analysis of data as it accumulates, which requires a planned error budget to keep the false-positive rate controlled.

References

1. Wilson, Probable Inference, the Law of Succession, and Statistical Inference (JASA 1927)
2. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology
3. Liang et al., Holistic Evaluation of Language Models / HELM (arXiv 2211.09110)
4. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
5. Cohen, A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement 1960)