Home

Offensive AI Security› Red-Team Measurement›The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a Grader

In this series

Red-Team Measurement

◆OverviewStart here
1What Is Attack-Success-Rate, Really? The Unit of Analysis, the Population, and the JudgeResearch
2The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a GraderResearch
3Sampling and Confidence: Wilson Intervals, Per-Family Estimation, and Sequential TestingResearch
4Transfer and Generalization: Measuring Whether an Attack Holds Across Models, Prompts, and TimeResearch
5A Reporting Standard: Datasets, Seeds, Judge, Confidence Intervals, and Threats to ValidityResearch

HostExpert

Murali Chillakuru

Press play to begin the conversation.

0:00

17:06

Red-Team Measurement · 2 / 5

The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a Grader