Research

Watch · narrated whiteboard episodesL3

Extraction Attacks: Stealing Models, Weights, and Training Data

A model exposed only through an API still leaks its parameters, its architecture, and its training data — and the leakage is quantifiable. This threat lab treats extraction as a measurement problem with information-theoretic limits: the query-access threat model, recovering a production model's last layer from logits, training-data memorization and its extraction rate, membership inference as a privacy metric, and the utility cost of every defense. Grounded in the primary extraction literature.

Murali Chillakuru·5 episodes
  1. 14 min Episode 1The Query-Access Threat Model: What a Black-Box Attacker Can and Cannot LearnA moderator and an expert map out what an attacker with only API access can extract from a model — and why the response type, not the network boundary, determines how much can be stolen.
  2. 14 min Episode 2Stealing the Last Layer: Recovering a Production Model's Final Projection from LogitsA moderator and an expert examine how the final linear map in a neural network — the projection from hidden states to outputs — can be recovered from a few thousand API responses, and what defending against this reveals about API design.
  3. 14 min Episode 3Training-Data Extraction and Memorization: Eidetic Memorization and the Extraction RateA moderator and an expert examine how large language models verbatim reproduce fragments of their training data, why scale and duplication amplify memorization, and how to measure and bound the extraction rate.
  4. 13 min Episode 4Membership Inference: Shadow Models, Loss Thresholds, and Calibrated AUC as a Privacy MetricA moderator and an expert examine how to decide whether a specific record was in a model's training set — and why the answer at the low-false-positive end of the ROC curve is the severity that matters.
  5. 15 min Episode 5Defenses and Their Costs: Truncation, Noise, Rate Limits, and Differential-Privacy AccountingA moderator and an expert price the extraction defense menu honestly — every control buys confidentiality with a currency, and choosing well means knowing exactly what you're spending.