Tracing, Metrics & Telemetry — Production LLM Eval & Observability

Software is invisible while it runs. You cannot watch a request move through a service the way you can watch a parcel move through a warehouse, so you need signals: small, structured facts emitted by the system itself. Tracing, metrics, and telemetry are the practice of collecting those facts, keeping their meaning intact, and using them to make better decisions under pressure.

Telemetry starts with one humble fact

Telemetry is data a system emits about its own behavior. The word comes from remote measurement: learning what is happening somewhere you cannot directly see. In software, that somewhere might be a server, a database, a background job, a mobile app, a model endpoint in Microsoft Foundry, or a sign-in flow using Microsoft Entra ID.

The smallest useful unit is an event: something happened at a time, in a place, with some context. A user clicked a button. A payment request timed out. A model returned an answer. A cache missed. Good telemetry preserves those facts in a form that humans and machines can search, count, compare, and connect.

Telemetry is not the same as monitoring. Telemetry is the raw and processed signal. Monitoring is the use of that signal to notice conditions that matter. Observability goes one step further: it is the ability to ask new questions about a system without changing the code first.

Telemetry is the raw signal; monitoring notices conditions; observability asks new questions.

The three everyday signals: metrics, traces, and logs

A metric is a number measured over time. How many requests per second? How long did they take? How much memory is used? Metrics are compact and fast to query, which makes them excellent for dashboards, alerts, and trend analysis. Their weakness is that a number often says what changed without explaining why.

A trace follows one unit of work as it crosses boundaries. A single user request might enter an application programming interface (API), call two services, query a database, and publish a message. A trace links those steps into one journey. Its strength is causality: it can show where time was spent and which dependency failed.

A log is a timestamped record, usually written as text or structured fields, about something the code chose to report. Logs are flexible and rich, but they can become noisy and expensive if every line is treated as equally important.

Telemetry is the umbrella over all three. Metrics, traces, and logs are not competitors. They answer different questions, and the best systems let you move between them: from an alerting metric, to a trace that shows the slow path, to a log entry that explains the exact error.

📌

Use the right signal. Metrics show that a pattern changed, traces show where a single journey went, and logs often explain the local detail.

Metrics count behavior, traces follow one request, and logs record what code chose to report.

A trace is a story made of spans

A trace represents one end-to-end operation. Inside it are spans. A span is one timed step in that operation: receive request, validate token, call inventory service, query database, render response. Each span has a start time, an end time, a name, and attributes that describe the work.

The important idea is context propagation. When one service calls another, it passes along a trace identifier so the next service can attach its span to the same trace. Without that shared identifier, the journey breaks into disconnected fragments. With it, you can see cause and effect across process, network, and team boundaries.

Distributed tracing matters because modern systems are distributed by default. Even a simple page load may involve identity, configuration, storage, search, payments, model inference, and external APIs. When something is slow, the user experiences one delay, but the system contains many possible sources.

One trace links timed spans as a request crosses services, sharing a trace identifier.

Metrics turn behavior into questions you can track

A metric has a name, a value, a time, and usually labels. A label is a small piece of context, such as region, service name, endpoint, or result code. Labels make metrics useful because they let you compare slices of behavior: errors by region, latency by endpoint, traffic by customer tier.

The most common service metrics are request rate, error rate, and latency. Rate asks how much work is arriving. Error rate asks how often work fails. Latency asks how long work takes. Together, these form a practical starting point because they describe demand, correctness, and speed.

The trap is cardinality. Cardinality means the number of distinct label combinations. A label such as region may have a handful of values. A label such as user identifier may have millions. High-cardinality metrics can become slow, costly, or impossible to store efficiently. Put unique identifiers in traces or logs, not in metric labels unless your platform is designed for it.

Use metrics for trends, alerts, and service level reporting.
Use stable labels such as service, region, operation, and outcome.
Avoid labels that create one time series per user, request, session, or document.
Prefer percentiles for latency because averages hide slow users.
Review old metrics and remove the ones nobody uses.

⚠️

Cardinality costs. A metric can look harmless in code but create millions of time series once labels are combined.

Instrumentation is design, not decoration

Instrumentation is the act of adding code or configuration that emits telemetry. It is tempting to treat it as a finishing touch, but the best time to design telemetry is when you design the feature. Ask what someone will need to know when the feature is slow, wrong, unavailable, or unexpectedly expensive.

Good instrumentation has consistent names, consistent units, and consistent context. If one team records duration in milliseconds and another in seconds, comparisons become unreliable. If one service calls an operation checkout and another calls the same operation purchase, dashboards and traces become harder to join.

OpenTelemetry is an open standard for collecting and describing telemetry across languages and vendors. Its value is not magic; its value is shared vocabulary. When libraries, services, collectors, and observability tools agree on trace identifiers, span fields, metric instruments, and semantic conventions, teams can change parts of the pipeline without rewriting every application.

Name the operation in language a human can understand.
Record the outcome, such as success, failure, cancellation, or timeout.
Capture duration with a clear unit.
Attach safe context, such as service, version, region, tenant tier, and dependency name.
Exclude secrets and personally identifiable information (PII) unless there is a strict, approved reason.

⚠️

Do not leak secrets. Telemetry often travels farther and lives longer than application memory, so never record passwords, tokens, keys, or unnecessary personal data.

From signal to action: the operating loop

Telemetry earns its keep when it changes behavior. A team observes a symptom, forms a question, investigates with signals, makes a change, and then checks whether the change helped. This loop is the practical heart of reliability work.

Alerts should represent user impact or a fast path to user impact. A server being busy is not automatically a problem; users failing to complete checkout is. A queue growing for ten seconds may be normal; a queue growing until work misses its deadline is not. Good alerts are few, meaningful, and tied to an owner.

A service level objective (SLO) is a target for acceptable service behavior, such as 99.9 percent of requests completing successfully within 500 milliseconds over 30 days. SLOs turn telemetry into a contract with reality. They help teams decide when to ship faster, when to pay down reliability risk, and when to stop and fix.

Teams observe a symptom, investigate with signals, make a change, and verify it helped.

The decisions telemetry should improve

For practitioners, telemetry improves everyday judgment. Which dependency is slow? Did the new release increase errors? Is one region behaving differently? Should this incident be rolled back, mitigated, or investigated further? The answer should come from evidence, not from the loudest guess in the room.

For leaders, telemetry connects engineering work to risk, cost, and customer experience. It shows whether reliability is improving, whether teams are spending too much time on preventable incidents, whether capacity is being wasted, and whether customers are receiving the experience the organization promised.

The hardest leadership decision is often not which tool to buy, but which questions to standardize. If every service reports success differently, the organization cannot compare reliability. If every team invents its own severity levels, incidents cannot be learned from consistently. Shared definitions make local autonomy safer.

What user journeys matter most?
What does good enough mean for each journey?
Which signals prove that users are succeeding or failing?
Who owns each alert and each service level objective?
What telemetry must be retained for learning, audit, or compliance?
What telemetry should be sampled or discarded to control cost?

✅

Measure what matters. The best telemetry program starts with the decisions people need to make, then works backward to the signals required.

Key takeaways

Telemetry is data a system emits about its own behavior; monitoring and observability are ways of using it.
Metrics show patterns over time, traces show end-to-end journeys, and logs provide detailed event records.
A trace is made of spans connected by shared context, which preserves causality across services.
Good metrics need careful labels because high cardinality can create large cost and performance problems.
Instrumentation should be designed with the feature, using consistent names, units, outcomes, and safe context.
Telemetry is most valuable when it supports real decisions about reliability, cost, risk, and customer experience.

Glossary

Telemetry: Data a system emits about its own behavior so people and tools can understand what is happening.
Metric: A numerical measurement recorded over time, often with labels that describe where or how it was measured.
Trace: A connected record of one operation as it moves through services, dependencies, and boundaries.
Span: One timed step inside a trace, such as a service call, database query, or model request.
Log: A timestamped record of an event or condition that code chose to report.
Cardinality: The number of distinct label combinations in a metric, which affects storage cost and query performance.
Instrumentation: The code or configuration that causes an application or platform to emit telemetry.
OpenTelemetry: An open standard for describing, collecting, and moving telemetry across languages, libraries, and tools.
Service level objective: A measurable target that defines acceptable service behavior over a stated period.