Offensive AI Security› Red-Team Measurement›The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a Grader

In this series

Red-Team Measurement

1What Is Attack-Success-Rate, Really? The Unit of Analysis, the Population, and the JudgeResearch
2The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a GraderResearch
3Sampling and Confidence: Wilson Intervals, Per-Family Estimation, and Sequential TestingResearch
4Transfer and Generalization: Measuring Whether an Attack Holds Across Models, Prompts, and TimeResearch
5A Reporting Standard: Datasets, Seeds, Judge, Confidence Intervals, and Threats to ValidityResearch

← All research

HostExpert

Murali Chillakuru

Press play to begin the walkthrough.

0:00

16:42

Red-Team Measurement · 2 / 5

The Judge Problem: LLM-as-Judge Bias, Human-Label Reliability, and Calibrating a Grader

Back