An Observability Reference Architecture: Sampling, Cost, Privacy, and the Feedback Loop

Individually, the components of AI observability are inert. A semantic trace with no store to hold it, an eval with no budget to run it, a drift alarm with nowhere to escalate - each is a capability without a system. The value appears only when they are wired together into a pipeline that captures the right telemetry, decides what to keep, spends its budget deliberately, protects the sensitive data it collects, and - most importantly - feeds what it learns back into the product. This article specifies that architecture as a reference: the components and their connections, the three cross-cutting concerns that make or break it in production (sampling, cost, privacy), and the feedback loop that is the entire reason to build it. It also takes seriously that the observability system is itself software with its own failure modes, and offers a ladder for adopting it a rung at a time.

From Signals to a System

A working observability system is a directed pipeline with a loop. Telemetry is emitted by the instrumented application, buffered and sanitized by a collector, persisted in a store, scored by an eval service, watched for drift by a detector, surfaced to humans for triage, and - the part that closes the loop - fed back to update baselines and to grow the datasets that gate the next release. Everything else in this article is a decision about how to run one of those stages well.

Naming the stages matters because each is a place where an architecture succeeds or fails independently. Under-instrument and the store holds blind spots; skip the collector's redaction and the store becomes a liability; run the eval service on everything and the cost is ruinous; never close the loop and the whole system is an expensive dashboard nobody acts on.

The reference architecture below is deliberately vendor-neutral and minimal: it is the set of stages you cannot omit, not a particular product. Start from the stages, then make the three cross-cutting decisions - what to sample, what it costs, and how to keep it private - that determine whether the system is affordable and safe.

Sampling Strategy: Head, Tail, and Keep-the-Weird

You cannot afford to keep every trace at full fidelity or to eval every response, so sampling is not optional - and how you sample determines what you can see. Head sampling decides at the start of a trace, before anything has happened, usually by a fixed probability. It is cheap and simple, but it is blind to outcome: it discards a rare catastrophic failure with the same probability as a boring success. Tail sampling decides after the trace completes, when the outcome is known, so it can keep the interesting ones - errors, high latency, low eval scores, unusual tool use - and downsample the routine.

The design rule is head-sample for volume and tail-sample for signal: apply a modest head sample to control raw ingestion, then a tail policy that retains anomalies at a far higher rate than nominal traffic. Evals follow the same logic - spend the judge budget preferentially on flagged, low-confidence, or novel outputs rather than uniformly, because a uniform online eval on one percent of traffic estimates an average while missing the tail that actually hurts.

Sampling is also a measurement decision, not only a budget one. An eval sample of size n estimates a rate with an uncertainty that shrinks only as one over the square root of n, so rare-but-severe failures need targeted retention to be seen at all.

A Cost Model

An observability system has three real spend drivers, and naming them makes the trade-offs explicit. Storage scales with retained volume times retention duration. The eval service scales with the online eval rate times the price of a judge call. Human review scales with the flag rate times the cost of a review. A compact model captures the total, and every term maps to a knob you actually control.

The knobs are exactly the levers of the previous section plus retention and model tier. Lower the tail-retention volume or shorten retention to cut storage; lower the online eval rate or use a cheaper judge tier for routine traffic (reserving a strong judge for flagged cases) to cut eval cost; raise alarm precision so fewer false flags reach a human to cut review cost. The point of writing the model down is that these are conscious trade-offs against detection quality, not incidental bills.

The table lists each driver with what it scales with and its primary knob, so a team can decide where to spend rather than discovering the cost after the fact.

The Privacy Plane

Semantic telemetry captures prompts, plans, and retrieved documents, so privacy is not a component bolted on at the end - it is a plane that cuts across every stage. The first control is redaction at emission: sensitive fields are detected and removed or tokenized inside the process, before telemetry ever reaches the collector, so the raw payload is never transmitted. The second is tiered retention: keep a small sample of raw content briefly for debugging, keep hashes and references for as long as you need to correlate, and let the raw tier expire fast.

Two more controls complete the plane. Access to captured reasoning must be governed by the same controls as the production data it came from - a trace store full of user prompts is not a lower-sensitivity system than the application. And data residency and retention obligations apply to telemetry exactly as they do to primary data. The NIST AI Risk Management Framework treats privacy as a core trustworthiness property, and the OWASP Top 10 for LLM Applications flags sensitive-information disclosure precisely because observability done naively becomes the leak.

The governing principle is that the telemetry must never be a softer target than the system it observes. If your application encrypts and access-controls user data, the trace store that now contains that same data inherits every one of those obligations.

The Feedback Loop

The reason to build any of this is the loop, and there are really two of them. The operational loop runs at incident speed: a drift alarm fires, a human or an automated triage step localizes it using the semantic traces, a fix or a rollback is applied, and the baseline is updated so the new normal is recognized. The improvement loop runs at release cadence: the traces you captured - especially the flagged and human-labeled ones - are curated into eval datasets, those datasets become an offline suite, and that suite gates the next release so the same regression cannot ship twice.

This is what turns observability from watching into learning. Without the improvement loop, every incident is rediscovered from scratch; with it, each labeled failure permanently raises the floor. It is also the concrete realization of continuous-monitoring doctrine: NIST SP 800-137 frames monitoring as an ongoing define-collect-analyze-respond-update cycle, and the feedback loop is where the update happens - baselines, thresholds, and eval sets all revised as the system and its usage move.

Designing for the loop changes what you instrument. You capture not only to see the present but to build the dataset that judges the future, which is why the flagged, calibrated cases from the drift ladder are the most valuable telemetry you own.

Failure Modes of the Architecture Itself

The observability system is software, and software fails - so design against its own failure modes. Blind spots: any stage a trace does not pass through is invisible, so instrument coverage is a first-class metric, not an assumption. The eval service as a single point of failure and a cost bomb: a synchronous judge on the hot path can add latency and, if unbounded, run up cost without limit - run it asynchronously, sample it, and cap it. Alert fatigue: an imprecise detector trains operators to ignore it, which is worse than no detector, so precision and aggregation are reliability features, not niceties.

Two failure modes are specific to this domain. The trace store is a breach target because it concentrates sensitive reasoning - the privacy plane is its mitigation, and it is not optional. And feedback poisoning: because the improvement loop turns captured traces into the eval set that gates releases, bad or adversarial labels can corrupt the very standard the system holds itself to - so curated eval data needs the same provenance, review, and access discipline as production code.

Each of these has a guard, and the guards are cheap relative to the failure. Coverage metrics, asynchronous capped evals, alarm precision and aggregation, the privacy plane, and provenance on eval data are the difference between an observability system you can trust and one that quietly becomes its own risk.

An Adoption Ladder

You do not build this all at once, and pretending otherwise stalls teams before they start. The ladder is incremental and each rung is independently useful. Rung one: emit semantic traces and the basic behavioral metrics - you can now see what the system did. Rung two: add reference-free evals on a sample and a real sampling policy - you can now estimate quality and afford to keep the right traces. Rung three: add drift detection with a false-alarm budget and the operational feedback loop - you can now catch regressions and act on them. Rung four: add the improvement loop, calibration, and the full privacy plane - you now learn from every incident and do it safely.

The ordering is not arbitrary: each rung produces the input the next one consumes. Traces feed evals; evals feed drift detection; drift detection feeds the feedback loop; the feedback loop feeds the datasets that make the whole thing improve. A team can stop at any rung and have a working system, then climb when the next problem earns the investment.

That incrementalism is the final design principle. Observability for non-deterministic AI is a substantial system, but it is not a monolith - it is a ladder of independently valuable capabilities, assembled from semantic traces, behavioral metrics, evals, drift detection, and a loop, protected by a privacy plane and paid for by a conscious cost model. Built that way, it turns an unpredictable system into an accountable one.

1. Rung 1 - Traces + behavioral metrics: see what the system did.
2. Rung 2 - Reference-free evals + sampling policy: estimate quality affordably.
3. Rung 3 - Drift detection + false-alarm budget + operational loop: catch and act on regressions.
4. Rung 4 - Improvement loop + calibration + privacy plane: learn from every incident, safely.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

sampling:
  head_rate: 0.25                 # cap raw ingestion
  tail_keep:
    error: 1.0                    # keep all errors
    eval_quality_below: 0.5       # keep low-scoring decisions
    novel_tool_call: 1.0          # keep unusual tool use
    default: 0.05                 # downsample routine traffic
retention:
  raw_content_days: 7             # short-lived raw tier
  hashes_and_refs_days: 180       # long-lived correlation tier
  eval_scores_days: 365
privacy:
  redact_at_emission: true
  access: same_as_source_data

on drift_alarm(signal):                 # OPERATIONAL loop
    cases = store.query(low_score_traces, window=recent)
    cause = triage(cases)               # localize via semantic traces
    apply_fix_or_rollback(cause)
    baselines.update(signal, new_reference_window)

nightly():                              # IMPROVEMENT loop
    labeled = human_review(store.sample(flagged))   # provenance-tracked
    eval_set.add(labeled)               # curated, reviewed
    report = offline_suite.run(candidate_release, eval_set)
    release_gate.require(report.quality >= baseline)

Glossary

Reference architecture

A vendor-neutral specification of the essential stages and connections of a system, independent of any particular product.

Head sampling

Deciding whether to keep a trace at its start by fixed probability; cheap but blind to the eventual outcome.

Tail sampling

Deciding whether to keep a trace after it completes, so anomalies (errors, low scores) can be retained preferentially.

Privacy plane

The cross-cutting set of controls - redaction, tiered retention, access control, residency - that protects captured telemetry.

Operational feedback loop

The incident-speed cycle of detect, triage, fix or roll back, and update baselines.

Improvement feedback loop

The release-cadence cycle that curates captured, labeled traces into eval datasets that gate future releases.

Feedback poisoning

Corruption of the eval datasets built from captured traces, via bad or adversarial labels, degrading the quality bar itself.

References

1. NIST AI 100-1, Artificial Intelligence Risk Management Framework (AI RMF 1.0)
2. NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative AI Profile
3. NIST SP 800-137, Information Security Continuous Monitoring (ISCM)
4. OWASP Top 10 for Large Language Model Applications (2025)
5. Liang et al., Holistic Evaluation of Language Models (HELM)
6. OpenTelemetry Semantic Conventions for Generative AI Systems
7. Basseville & Nikiforov, Detection of Abrupt Changes: Theory and Application (1993)