Research seriesL2data science
Traditional APM assumes deterministic services; autonomous AI systems are not. This series builds runtime observability for non-deterministic agents from the ground up: why classic monitoring fails, traces + metrics + evals as first-class telemetry, semantic capture of reasoning and tool calls (OpenTelemetry GenAI), detecting regressions without ground truth, and a cost-, privacy-, and sampling-aware reference architecture. Grounded in NIST AI RMF, NIST SP 800-137, OWASP LLM Top 10, the OpenTelemetry GenAI conventions, and the evaluation-methodology literature.
APM assumes determinism, a binary success oracle, and bounded cardinality. Autonomous AI breaks all three - here is why observability must be rebuilt.
Classic observability rests on logs, metrics, and traces - none of which carry correctness. For AI the load-bearing triad becomes traces, metrics, and evals.
A trace with only timing is syntactic. Semantic telemetry records meaning - plan, tool I/O, and the evidence that grounded each claim - so a decision is auditable and replayable.
In production there is no answer key. You cannot measure accuracy - so you detect that behavior changed, from proxy signals, and infer whether it got worse.
Signals only matter once they are assembled into a system - with a sampling strategy, an honest cost model, a privacy plane, and a loop that turns alarms back into fixes and eval data.