Throughput, Latency & SLO Trade-offs — LLM Inference Optimization & Serving

Every digital system answers three practical questions: how many things can it handle, how long does each thing take, and how sure are we that users will get an acceptable result. Throughput, latency, and service-level objectives are the names we give those questions. They are simple ideas, but they interact in ways that surprise teams: making more work fit can make individual requests slower, chasing the fastest possible response can waste capacity, and promising perfection can force expensive designs that still fail in different ways.

Start with work, not machines

A system is a place where work enters, waits if necessary, gets served, and leaves. The work might be a web request, a payment, a model inference, a file upload, or a database query. The machines, networks, and software matter, but the first-principles view is simpler: items arrive, resources process them, and completed items depart.

Throughput is the rate at which completed work leaves the system. Latency is the time one item spends from start to finish. If a restaurant serves 120 meals per hour, that is throughput. If your meal takes 18 minutes from ordering to arrival, that is latency. Both numbers describe the same operation from different angles.

A useful system must care about both. High throughput with painful latency feels like a crowded train station. Low latency with tiny throughput feels like a private elevator: pleasant for one person, useless for a crowd.

Work arrives, waits in a queue if needed, is served by a resource, and departs when complete.

Latency is a distribution, not a single number

Latency varies because real systems vary. Some requests are easy, some need more data, some hit a warm cache, some wait behind other work, and some meet a slow network hop. A single average can hide the experience of many users.

Percentiles are often better than averages. The 50th percentile, also called the median, is the point where half of requests are faster and half are slower. The 95th percentile means 95 percent of requests are faster than that value and 5 percent are slower. The 99th percentile is often called P99 and describes the tail of the experience: the slow cases that can dominate complaints, retries, and support tickets.

The tail matters because users do not experience averages. A page that usually loads in 300 milliseconds but sometimes takes 8 seconds may feel unreliable even if the average looks healthy. In systems with many steps, tail latency compounds: if a single user action calls ten services, the chance that at least one step is slow can be high.

⚠️

Beware averages. An average latency target can pass while a meaningful share of users still has a bad experience.

Latency is a distribution: the median is typical, while P95 and P99 describe the slow tail.

Throughput rises until something saturates

Every system has a limiting resource. It may be central processing unit capacity, memory, disk input and output, network bandwidth, database locks, a thread pool, a rate limit, or an external dependency. Throughput rises as more work arrives only while there is spare capacity. Once the limiting resource is busy enough, extra work mostly becomes waiting.

Saturation is the condition where a resource is used so heavily that it cannot absorb normal variation. A server at 40 percent utilization may handle bursts cleanly. The same server at 90 percent may look efficient until a small surge causes queues, timeouts, and retries. High utilization is not automatically bad, but high utilization without headroom is fragile.

The important relationship is this: as utilization approaches the limit, latency usually grows faster than throughput. Teams often discover this during launches or seasonal peaks. The system is still completing work, so throughput appears acceptable, but users wait longer because each new request joins a longer line.

Queues are where the trade-off becomes visible

A queue is a waiting line. Queues are not failures; they are a normal way to smooth bursts when work arrives faster than resources can serve it for a short time. Message brokers, web servers, database connection pools, and background job systems all use queues in different forms.

The danger is an unbounded queue. If arrivals keep exceeding service capacity, the queue grows, latency grows, and eventually something breaks. Users may retry, which creates more arrivals and makes the queue even longer. This is how a small slowdown can become a large incident.

Back pressure is the practice of pushing back when a system is near its limit. That pushback may be a clear error, a retry-after response, slower intake, load shedding, or a smaller batch size. Back pressure can feel harsh because some work is delayed or rejected, but it protects the system from letting every user suffer slowly.

⚠️

Retry storms. Retries without limits can turn a temporary slowdown into a self-inflicted outage.

Service-level objectives make trade-offs explicit

A service-level objective, or SLO, is a measurable target for user-facing behavior. A latency SLO might say that 95 percent of search requests should complete within 400 milliseconds over 30 days. An availability SLO might say that 99.9 percent of requests should return a valid response in a month.

A good SLO has four parts: what is measured, which users or requests count, the target threshold, and the measurement window. Without those parts, the promise is too vague to guide engineering decisions. Fast is not an SLO. P95 checkout latency under 700 milliseconds for successful user checkouts over a rolling 28-day window is much closer.

SLOs are useful because they define acceptable imperfection. A 99.9 percent target allows about 0.1 percent bad events in the chosen window. That allowance is called an error budget. If the system is comfortably inside the budget, a team may choose to ship changes faster. If the budget is burning quickly, the team may slow releases and focus on reliability work.

A service-level agreement, or SLA, is different. An SLA is usually a formal commitment to a customer, often with business consequences if it is missed. An SLO is an internal or product-level operating target. The healthiest pattern is to set SLOs tighter than SLAs so teams can react before customers are owed remedies.

Measure what users feel, not only what servers report.
Use percentiles for latency so tail behavior is visible.
Choose a window long enough to smooth noise but short enough to act.
Keep the target realistic enough that teams can make informed trade-offs.

A good service-level objective names what is measured, which requests count, the target, and the window.

The real decisions are business decisions

Throughput, latency, and SLOs are technical measurements, but the choices around them are leadership choices. The central question is not how fast can we make it. The better question is how fast does this need to be, for whom, at what cost, and with what consequence if we miss.

A payments flow, an emergency dispatch system, and a nightly analytics report should not have the same latency target. The payment may need low tail latency and strong availability because delay can lose revenue and trust. The dispatch system may justify expensive redundancy because human safety is involved. The analytics report may accept minutes of delay if that saves money and simplifies operations.

Practitioners face concrete design choices. Should requests be handled synchronously or placed on a queue? Should capacity be reserved for priority customers? Should the system reject excess work early or allow long waits? Should model calls in Microsoft Foundry be cached, batched, streamed, or degraded to a smaller model when demand spikes? None of these has a universal answer; the SLO gives the team a way to choose deliberately.

📌

Ask the cost question. The right target is the one whose user value is worth the engineering, operational, and financial cost.

Tune the system with measurements, not guesses

Improvement starts with instrumentation: collect request counts, latency percentiles, error rates, saturation signals, queue depth, and retry counts. Look at the system by user journey, endpoint, customer segment, region, dependency, and time of day. Aggregate numbers are useful for steering, but incidents often hide in one path or one tenant.

When a target is missed, separate causes before changing architecture. Is work arriving faster than expected? Is service time too slow? Is a dependency adding tail latency? Are retries multiplying load? Is a cache cold? Is one expensive request type sharing capacity with many cheap ones? Each cause suggests a different remedy.

Common remedies include adding capacity, reducing work per request, caching repeated answers, batching small operations, separating priority traffic, setting timeouts, limiting retries, using back pressure, and moving non-urgent work to background processing. The best teams test these changes against the SLO, not against a vague hope that the system feels faster.

The final discipline is to keep SLOs alive. Review them when product behavior changes, when customer expectations change, when traffic grows, and when new dependencies are added. An SLO is not a plaque on the wall; it is a steering instrument.

Define the user journey and success condition.
Measure throughput, latency percentiles, errors, and saturation.
Find the limiting resource or dependency.
Choose one change that addresses the actual limit.
Verify the result against the SLO.

✅

Simple loop. Measure, explain, change, and verify is usually more effective than redesigning from intuition.

Key takeaways

Throughput is the rate of completed work; latency is the time one item takes from start to finish.
Latency should be viewed as a distribution, because tail percentiles often explain user pain better than averages.
As a system nears saturation, queues grow and latency can rise sharply even if throughput still looks healthy.
SLOs turn vague wishes into measurable targets with explicit error budgets.
Back pressure, timeouts, and retry limits protect overloaded systems from collapse.
The right target depends on user value, risk, cost, and the consequence of failure.

Instrument the system, diagnose the real cause, apply a remedy, and verify it before repeating.

Glossary

Throughput: Throughput is the rate at which a system completes work over a period of time.
Latency: Latency is the time one item of work takes from the start of a request to its completion.
Percentile: A percentile is a point in a set of measurements below which a given percentage of results fall.
P99: P99 is the 99th percentile, meaning 99 percent of measured requests were faster and 1 percent were slower.
Saturation: Saturation is the state where a resource is so heavily used that additional work mostly creates waiting.
Queue: A queue is a waiting line for work that has arrived but cannot yet be served.
Back pressure: Back pressure is a mechanism that slows, rejects, or redirects incoming work when a system is near its limit.
Service-level objective: A service-level objective is a measurable target for the behavior users should receive from a service.
Error budget: An error budget is the amount of failure or slowness allowed by a service-level objective during a measurement window.