Transfer and Generalization: Measuring Whether an Attack Holds Across Models, Prompts, and Time

An attacker tunes a jailbreak until it reliably defeats a particular model on a particular set of prompts, then reports the resulting rate as the attack's strength. The number is real but it answers the wrong question. Just as a classifier's accuracy on its own training data tells you almost nothing about how it will perform in the wild, an attack's success on the very targets it was optimized against tells you almost nothing about whether it generalizes. The scientifically meaningful quantity is transfer: does the attack still work on a model it was not tuned on, on prompts it has not seen, and on a version released after it was built? This article treats those questions as measurement problems and builds the protocol to answer them honestly.

A development score is not a generalization claim

Supervised learning learned this lesson long ago: performance measured on the data used to fit a model is optimistically biased, sometimes wildly so, which is why the field separates training, validation, and test sets and reports only the last. Attack development is structurally identical. An adversarial suffix, a jailbreak template, or a poisoning trigger is fit — by hand or by optimization — against feedback from specific targets. Evaluating it on those same targets measures how well it memorized them, not how far it reaches.

Holistic evaluation practice makes the same point for benchmarking language models generally: a result is only informative about deployment if the evaluation covers scenarios the system was not tuned to, and if the metric is reported across a breadth of conditions rather than a single favorable one. The generalization gap — the difference between development-set and held-out performance — is the quantity that separates a real capability from an artifact of tuning.

The remedy is to define, before measuring, what the attack must generalize across, and to hold those things out. Everything in this article follows from taking that separation seriously for attacks the way the field already takes it seriously for models.

Three axes an attack must generalize across

Attack generalization is not one question but three, and they fail independently. The model axis asks whether an attack tuned on one target still works on a different model — a different architecture, scale, or vendor. The prompt axis asks whether it works on task instances it was not fit against, rather than the specific prompts used during tuning. The time axis asks whether it survives model updates: providers retrain and patch, so an attack's efficacy is non-stationary and can decay after any release.

These axes matter because a claim of the form "this attack works" implicitly quantifies over all three, while a development score quantifies over none. An attack can transfer across models yet collapse on new prompts, or work today and vanish next month. Reporting a single number without saying which axes were held out conflates three different generalizations into one, and the reader cannot tell which were actually tested.

The discipline is to name the axes explicitly and measure each. A transfer claim is only as strong as the axis that was held out most weakly.

Measuring cross-model transfer

Cross-model transfer is naturally expressed as a matrix. Index the rows by the source model an attack was tuned on and the columns by the target model it is evaluated against; each cell holds the success rate of that source's attack on that target. The diagonal is the development score — attack and evaluation on the same model — and is expected to be high. The off-diagonal cells are the transfer scores, and they are the honest measure of reach.

Reading the matrix disciplines the claim. A high diagonal with a low off-diagonal means the attack is target-specific and does not generalize across models, however impressive its headline number. A uniformly high matrix means the attack exploits something shared across targets — evidence of a common vulnerability rather than a bespoke trick. The gap between the diagonal and the off-diagonal is a direct estimate of the model-axis generalization gap.

Each cell is itself an estimate with sampling uncertainty and a judge, so the matrix is a grid of intervals, not point values. Reporting only the best off-diagonal cell — cherry-picking the target where transfer happened to land — is the transfer-matrix version of measuring on the training set.

Prompt and temporal generalization

The prompt axis is measured the same way the model axis is: hold out a set of task instances the attack was never tuned against and evaluate on them. If an attack was fit against ten harmful requests, its rate on those ten is a development score; its rate on a disjoint, comparably drawn set of requests is the prompt-generalization score. When the two diverge, the attack has overfit to the surface form of its tuning prompts rather than exploiting a durable mechanism.

The time axis is subtler because it changes underneath you. Providers retrain, fine-tune for safety, and patch specific exploits, so an attack's success rate is non-stationary — a function of the model version, not a constant. An efficacy number carries an implicit timestamp, and a rate measured against last quarter's version may not describe today's. Measuring temporal generalization means re-evaluating the frozen attack against successive releases and reporting the decay curve, not a single stale point.

Both axes share a moral: generalization is only demonstrated against data the attack could not have adapted to, whether that data is separated by prompt identity or by release date.

A held-out evaluation protocol for attacks

The protocol borrows directly from supervised learning. Partition the attack surface up front: a development split on which the attack may be tuned freely, and a test split — held-out models, held-out prompts, and a future model version — that is touched exactly once, at the end, to produce the reported numbers. Any information that flows from the test split back into tuning contaminates it and reinflates the development bias the split was meant to remove.

Concretely: freeze the attack after development; evaluate it on the held-out targets and prompts; and, where possible, re-evaluate on a later model release to estimate temporal decay. Report the development score and the held-out scores side by side so the generalization gap is visible. The single most important rule is that the test split is used once — repeatedly tuning against it turns it into a second development set and the guarantee evaporates.

This is unglamorous discipline, but it is the difference between a transfer claim and a memorization score dressed as one. The protocol is cheap relative to the credibility it buys.

Overfitting to a benchmark

Even a held-out protocol degrades if a fixed public benchmark is reused as the test set across many iterations. Each time results on that benchmark guide the next attack, the benchmark leaks into development and stops measuring generalization — a direct instance of the principle that a measure used as a target ceases to be a good measure. Over a community's many attempts, published rates on a popular jailbreak benchmark drift upward not because attacks generalize better but because the benchmark has been collectively overfit.

Contamination compounds the problem: if a benchmark's contents appear in training data or are otherwise memorized by the target, evaluating on it measures recall, not vulnerability. Holistic evaluation guidance responds by favoring broad, refreshed, and multi-scenario measurement over a single fixed leaderboard, precisely so that no one artifact can be gamed. For attack measurement the analog is to rotate held-out sets, keep a truly untouched test partition, and treat any long-lived public benchmark as a development aid, not a generalization oracle.

The defensive posture is humility about any single benchmark number. A rate that only exists on one heavily-used dataset is a claim about that dataset, not about attacks in general.

Threats to validity, and what to report

The threats here are generalization threats. Reporting a development score as a capability ignores the generalization gap entirely. Testing only the model axis while implying all three overstates reach. Cherry-picking the best transfer cell, the best prompt set, or the most favorable model version manufactures a number the data do not support. And reusing a public benchmark as a de facto training signal quietly destroys its ability to measure generalization at all.

The reportable package is a transfer profile, not a scalar: the development score; held-out scores on the model, prompt, and time axes, each with its interval and judge; the source-by-target matrix rather than a single transfer number; and the benchmark's provenance and reuse history. A rate accompanied by that profile states where the attack was shown to reach and where it was not. A lone percentage states neither.

As with every binary AI-security outcome, the statistics are indifferent to the attack's nature — extraction, evasion, poisoning all generalize or fail to along the same three axes. Hold out what the claim quantifies over, or the claim is only about the data you tuned on.

1. Separate a development split from a held-out test split before tuning anything.
2. Report the source-by-target transfer matrix, not a single transfer number.
3. Measure all three axes — model, prompt, time — and label which were held out.
4. Treat any reused public benchmark as a development aid, never a generalization oracle.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE / MOCK ONLY — no real target, no attack content
function transfer_profile(attack, dev, heldout):
    dev_score = evaluate(attack.tune_on(dev), dev)     # optimistic, expected high
    attack.freeze()                                    # no further tuning
    profile = {}
    for axis in ['models', 'prompts', 'versions']:
        profile[axis] = evaluate_with_interval(attack, heldout[axis])
    matrix = [[evaluate_with_interval(attack_from(s), t)
               for t in heldout['models']] for s in heldout['sources']]
    return Report(dev=dev_score, held_out=profile, transfer_matrix=matrix)

Glossary

Generalization gap

The difference between an attack's development-set success and its success on held-out data.

Transfer

The degree to which an attack tuned on one target succeeds on models, prompts, or versions it was not tuned on.

Transfer matrix

A source-by-target grid of success rates whose diagonal is development performance and off-diagonal is cross-model transfer.

Non-stationarity

The property that an attack's success rate changes over time as the target model is retrained or patched.

Held-out split

Models, prompts, or versions deliberately excluded from tuning and used once to measure generalization.

Benchmark overfitting

The upward drift of results when a fixed benchmark is repeatedly used to guide development, destroying its generalization signal.

Contamination

The presence of benchmark content in a model's training data, turning evaluation into a test of recall rather than vulnerability.

References

1. Liang et al., Holistic Evaluation of Language Models / HELM (arXiv 2211.09110)
2. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology
3. Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv 2306.05685)
4. Wilson, Probable Inference, the Law of Succession, and Statistical Inference (JASA 1927)
5. Cohen, A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement 1960)