Research

Watch · narrated whiteboard episodesL2

The Detection Limits of Autonomous SOC

The industry assumption that AI-based detection is uniformly better than signatures is partly correct and partly dangerous. Any anomaly detector operating at production scale faces an irreducible error floor set by the ROC trade-off and base-rate reality. Detection models age against evolving attackers — concept drift is not a software bug, it is a structural property of the adversarial environment. Shared detection models in multi-tenant SOC platforms leak behavioral signals across customers. Correlation-based triage is structurally blind to patient attackers who operate below the temporal window. And the deepest problem: you cannot build a ground-truth test set for the attacks you have missed. This series examines each limit formally, quantifies its cost in production settings, and builds the measurement discipline needed to reason honestly about what autonomous detection can and cannot do. Grounded in the anomaly-detection literature, adversarial ML research, NIST AI RMF, and MITRE ATLAS.

Murali Chillakuru·5 episodes
  1. 21 min Episode 1Statistical Limits of Autonomous Anomaly DetectionWhy every anomaly detector has an error floor, why rare attacks turn accurate alerts into mostly false alarms, and how autonomous response must price each action and make mistakes cheap to undo.
  2. 6 min Episode 2Concept Drift in Production Detection Models: The Defender's Red QueenEvery anomaly detector has an irreducible error floor — understanding the ROC trade-off, base-rate problem, and cost-sensitive thresholds determines which autonomous actions are safe to deploy.
  3. 5 min Episode 3Cross-Tenant Leakage in Shared Detection ModelsWhen many organizations share a single detection model trained on their collective telemetry, behavioral signals about one organization can be inferred by querying the model from another.
  4. 5 min Episode 4The Causal Gap: Why Correlation-Based Triage Fails Low-and-Slow AttacksAlert correlation that relies on temporal and statistical co-occurrence cannot detect an attacker who deliberately spaces actions to fall outside the correlation window — a structural blind spot.
  5. 7 min Episode 5Evaluating Detection Fidelity Without Ground TruthYou cannot build a labeled test set for what you have missed — but you can construct structured partial evaluation coverage through adversarial stress-testing, recall proxies, and red-team trace replay.