Research

Watch · narrated walkthroughs

Runtime Observability for Non-Deterministic AI

Traditional APM assumes deterministic services; autonomous AI systems are not. This series builds runtime observability for non-deterministic agents from the ground up: why classic monitoring fails, traces + metrics + evals as first-class telemetry, semantic capture of reasoning and tool calls (OpenTelemetry GenAI), detecting regressions without ground truth, and a cost-, privacy-, and sampling-aware reference architecture. Grounded in NIST AI RMF, NIST SP 800-137, OWASP LLM Top 10, the OpenTelemetry GenAI conventions, and the evaluation-methodology literature.

Murali Chillakuru·5 episodes
  1. 1
    The Observability Gap: Why Traditional APM Fails on Non-Deterministic AI Systems
    Watch ▸11:56
  2. 2
    The Pillars for AI: Traces, Metrics, and Evals as First-Class Telemetry
    Watch ▸11:44
  3. 3
    Semantic Telemetry: Capturing Reasoning, Tool Calls, and Decision Provenance with OpenTelemetry GenAI
    Watch ▸9:32
  4. 4
    Detecting Regressions Without Ground Truth: Reference-Free Evals and Drift Signals
    Watch ▸10:02
  5. 5
    An Observability Reference Architecture: Sampling, Cost, Privacy, and the Feedback Loop
    Watch ▸9:49