Research

Watch · narrated walkthroughs

Measuring Attack Success: A Data-Science Methodology for Red-Teaming

Most reported jailbreak rates are anecdotes, not measurements. This series builds a rigorous methodology for quantifying AI attack success: defining attack-success-rate, calibrating the judge, sizing samples with real confidence intervals, measuring transfer, and a reproducible reporting standard. Grounded in NIST AI 100-2, HELM, and the statistics primary literature.

Murali Chillakuru·5 episodes
  1. 1
    What Is Attack-Success-Rate, Really? The Unit of Analysis, the Population, and the Judge
    Watch ▸19:42
  2. 2
    The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a Grader
    Watch ▸16:42
  3. 3
    Sampling and Confidence: Wilson Intervals, Per-Family Estimation, and Sequential Testing
    Watch ▸15:41
  4. 4
    Transfer and Generalization: Measuring Whether an Attack Holds Across Models, Prompts, and Time
    Watch ▸14:03
  5. 5
    A Reporting Standard: Datasets, Seeds, Judge, Confidence Intervals, and Threats to Validity
    Watch ▸12:43