Training-Data Extraction and Memorization: Eidetic Memorization and the Extraction Rate

A model that has read a sentence enough times, or a rare enough sentence even once or twice, can reproduce it word for word when prompted the right way. Carlini and colleagues demonstrated that specific training examples — including personal information and unique strings — can be extracted from a deployed language model, and later work turned this into a measurement: what fraction of training data is memorized, and how that fraction scales. This is a confidentiality and privacy exposure distinct from stealing the model itself; the target is the data the model learned from. This article defines memorization rigorously, explains the extraction attack, treats the extraction rate as the measurable quantity it is, and pairs each driver with the training-time defense that reduces it.

Memorization versus generalization

A model generalizes when it captures patterns that apply beyond any single training example; it memorizes when it stores a specific example closely enough to reproduce it. These are not the same, and the distinction is what makes extraction a privacy problem rather than a capability. A generalized fact — that capital cities exist — reveals nothing about any particular training document; a memorized sequence — a specific person's contact details copied from a scraped page — reveals exactly that document. Extraction targets the second.

The sharpest definition is verbatim, eidetic recall: a sequence is memorized if the model reproduces it exactly, and it is eidetic to a degree that reflects how few times it appeared in training. A string memorized despite occurring only once or twice is especially concerning, because it means rarity is no protection — the very uniqueness that makes a secret sensitive can coincide with the model having stored it. Memorization is therefore not limited to common boilerplate; it reaches the rare, identifying strings privacy most cares about.

For the defender, the key implication is that memorization is a property of the trained weights, fixed at training time. No amount of API-side disclosure control removes a memorized sequence from the model; it only affects how easily the sequence can be surfaced. The durable controls therefore belong to the training pipeline, a theme every defense in this piece returns to.

The extraction attack: generate, then verify

The attack has two stages. First, generation: sample many continuations from the model, optionally seeded with plausible prefixes, to produce a large pool of candidate sequences that may include memorized text. Second, verification: score the candidates to find the ones the model treats as unusually likely or confident — memorized sequences tend to be assigned high probability relative to their genuine novelty — and keep those as probable training data. The attacker does not need the training set to run this; the model's own confidence is the signal that separates memorized from invented text.

The subtlety is the verification step, because a model assigns high probability to fluent generic text too. Effective extraction distinguishes memorized sequences from merely fluent ones using signals like the gap between the model's confidence and a reference expectation, or agreement across sampling conditions. This is a measurement problem: the attacker is estimating which candidates are memorized, with false positives (fluent non-memorized text) and false negatives (memorized text not surfaced) to control.

For the defender, the two-stage structure exposes two intervention points: make memorized sequences less likely to be generated (training-time deduplication and privacy), and make the confidence signal less exploitable (output filtering and score corruption). But the root cause — the sequence being memorized at all — is only addressed at the first.

What drives memorization

Quantifying memorization reveals it is not random but driven by measurable factors. Scale is the first: larger models memorize more of their training data, with the memorized fraction rising as capacity grows. Duplication is the second and strongest lever: a sequence that appears many times in the training corpus is far more likely to be memorized than a unique one, and the relationship is steep — repeated data is memorized data. Context length is the third: longer prompts give more room to elicit a memorized continuation, so more of what was stored becomes reachable.

These drivers are actionable precisely because they are measurable and, in the case of duplication, controllable. The dependence on duplication means that deduplicating the training corpus — removing near-identical copies — directly reduces the memorized fraction, which is why it is the single most effective training-time defense. Scale is harder to trade against, since capability rises with it, and context length is a serving choice with its own utility value.

The picture that emerges is a small set of knobs: reduce duplication aggressively, weigh the memorization cost of scale, and recognize that long-context serving expands what can be surfaced. Each is a lever a training and serving team can actually turn.

Quantifying the extraction rate

The extraction rate — the fraction of training data an attacker can recover — is a measurement with its own subtleties, and treating it rigorously avoids both alarmism and complacency. A useful distinction is between discoverable memorization, sequences that can be elicited when you already know a prefix, and extractable memorization, sequences an attacker can surface without prior knowledge. The former upper-bounds the latter; a model may have memorized far more than a knowledge-free attacker can practically pull out, so the honest risk figure is the extractable rate, not the total memorized fraction.

Measuring it well means specifying the attack budget, the verification judge and its error, and the population of sequences considered — the same measurement discipline any success-rate claim requires. An extraction rate reported without the number of queries, the verification method, and its false-positive rate is uninterpretable, and comparisons across models are only valid under matched conditions. This is a data-science problem wearing a security costume.

The practical value of quantification is that it lets a provider track the exposure over training choices: measure the extractable rate before and after deduplication, or across scales, and see the effect. Defense becomes an experiment with a metric, not a hope.

The privacy and confidentiality impact

The impact is concrete because the memorized content is real training data. Extracted sequences have included personal information, unique identifiers, and verbatim passages from source documents, so extraction is simultaneously a privacy violation (exposing individuals' data), a confidentiality breach (leaking proprietary or licensed text), and a compliance problem (surfacing data subject to legal protection). For a model trained on scraped or sensitive corpora, the memorized fraction is a latent disclosure liability baked into the weights.

This distinguishes extraction from the other attacks in the extraction family: the target is not the model's value but the data subjects' and content owners' rights. That changes who is harmed and which obligations apply — data-protection duties, breach-notification thresholds, and licensing terms — and it means the risk cannot be waved away as merely a competitive concern. A single extracted personal record can be a reportable event.

The proportionate response ties defense to the sensitivity of the training corpus. A model trained on public, low-sensitivity data carries less extraction risk than one trained on personal or proprietary data, and the strength of the training-time defenses should scale with that sensitivity. Knowing what went into training is the precondition for sizing the controls.

Defenses that follow from the mechanism

Because memorization is set at training time, the strongest defenses are there. Deduplication is first and most effective: removing near-duplicate sequences from the corpus directly lowers the memorized fraction, targeting the strongest driver. Differentially private training adds calibrated noise to the training process so that no single example can strongly influence the model, providing a formal bound on how much any one record is memorized — at a measurable utility cost. Together these attack the root cause: the sequence being stored at all.

Serving-time defenses complement but cannot replace them. Output filtering can block returns that match known-sensitive patterns or planted canaries; confidence-signal corruption makes the verification step harder; and query monitoring can flag the high-volume, structured sampling extraction requires. These raise the cost of surfacing memorized data but do not remove it from the weights, so they are a second line behind the training-time controls.

The measured posture is to deduplicate aggressively, apply differentially private training where the corpus is sensitive enough to justify its utility cost, and layer output filtering and monitoring at serving. Track the extractable rate as the metric that tells you whether the stack is working.

Limits and honest framing

The limits cut both ways. For the attacker, the extractable rate is far below the total memorized fraction, verification is noisy, and knowledge-free extraction is expensive — so the practical exposure, while real, is smaller than the scariest total-memorization numbers suggest. Over-stating extraction as trivial mass exfiltration misrepresents the risk. For the defender, deduplication reduces but does not eliminate memorization, differentially private training costs utility that some applications cannot afford, and output filtering cannot anticipate every sensitive string. Each control is partial.

There are honest tensions too. Scale drives both capability and memorization, so the most useful models are also the most memorizing, and long context is valuable yet expands elicitation. These trade-offs mean memorization cannot be driven to zero without giving up capability, so the goal is proportionate reduction matched to corpus sensitivity, not elimination. Certification-style guarantees exist only through differentially private training, at a cost.

Held honestly, the measurement framing is the contribution: memorization is a quantifiable, training-time property with identifiable drivers and matched defenses, and the extractable rate is the metric that turns the problem from anecdote into engineering. Deduplicate, privatize where warranted, filter and monitor at serving, and measure the extractable rate to know if it is working.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

# DEFENSIVE / MOCK ONLY — measures own-canary extractability, not real user data
function measure_extractability(model, planted_canaries, budget):
    surfaced = 0
    for _ in range(budget):
        cand = model.sample()                     # generate candidates
        if verify_memorized(cand, reference):     # confidence gap vs reference
            if cand in planted_canaries:
                surfaced += 1
    return Report(extractable_rate=surfaced / len(planted_canaries),
                  budget=budget, verifier='confidence-gap',
                  note='track across dedup / DP settings')

Glossary

Memorization

A model's storage of specific training examples closely enough to reproduce them, as opposed to learning patterns.

Eidetic memorization

Verbatim recall of a sequence that appeared few times in training, so rarity does not prevent it.

Generate-then-verify

The extraction method of sampling many candidate sequences then keeping those confidence marks as memorized.

Discoverable memorization

Memorized sequences elicited when a prefix is already known; an upper bound on extractable memorization.

Extractable memorization

Memorized sequences a knowledge-free attacker can surface, the honest risk figure.

Deduplication

Removing near-duplicate training sequences to reduce the memorized fraction, the strongest training-time defense.

Differentially private training

Training with calibrated noise so no single example strongly influences the model, bounding memorization.

References

1. Carlini et al., Extracting Training Data from Large Language Models (arXiv 2012.07805)
2. Carlini et al., Quantifying Memorization Across Neural Language Models (arXiv 2202.07646)
3. NIST AI 100-2 e2023 — Adversarial Machine Learning: A Taxonomy and Terminology
4. Shokri et al., Membership Inference Attacks Against Machine Learning Models (IEEE S&P 2017)
5. Tramèr et al., Stealing Machine Learning Models via Prediction APIs (USENIX Security 2016)