The Decoding Bottleneck — Speculative Decoding & KV-Cache

A large language model can read a long prompt in one burst, then seem to slow down when it writes the answer. That is not a paradox. Reading and writing use the machine in different ways. The writing step, called decoding, must usually choose one small piece of text, then use that choice to choose the next one, and so on. This serial rhythm is the decoding bottleneck: the point where an otherwise powerful system is constrained by the need to produce output in order.

Start with the simplest machine

Imagine a machine that completes sentences. It does not write a whole paragraph at once. It looks at the text so far and asks one question: what is the next most likely piece of text? After it picks that piece, the text has changed, so it asks the same question again. The answer is built one step at a time.

Those pieces are called tokens. A token may be a word, part of a word, punctuation, or a space-like fragment. The model does not think in pages or sentences; it receives and emits tokens. Decoding is the process of turning the model's next-token predictions into actual output tokens.

The bottleneck appears because token 57 in the answer depends on token 56. You cannot fully compute token 57 before deciding token 56, because the model's input will be different depending on what token 56 is. This dependency limits how much of the output can be created in parallel.

Prefill is broad; decoding is narrow

A large language model (LLM) request has two very different phases. In the prefill phase, the model reads the input prompt. If the prompt is long, there is a lot of work to do, but much of it can be arranged as large blocks of arithmetic. Modern graphics processing units (GPUs) are very good at that kind of work.

In the decoding phase, the model writes the answer. Each new token requires another pass through much of the model. The work per pass may be smaller than prefill, but the passes are chained. The hardware may have thousands of compute units available, yet the request advances by one generated token at a time.

This is why a system can have a long delay before the first token when the prompt is large, and still have a separate speed limit once the answer begins streaming. These are related but distinct experiences: time to first token and tokens per second.

📌

Two clocks. A slow first token and slow streaming speed have different causes, so they often need different fixes.

Prefill reads the prompt in parallel blocks while decoding writes one token per pass.

The model is not just computing; it is remembering

During generation, the model must keep track of what it has already processed. In transformer models, a common structure stores intermediate attention data in a key-value (KV) cache. The cache prevents the model from recomputing the entire past from scratch for every new token.

The KV cache is helpful, but it is not free. It consumes memory, and each decoding step must read from it. As the conversation grows, the cache grows too. A longer context means more memory pressure and more data movement.

This matters because decoding is often limited less by raw arithmetic and more by moving data to the right place at the right time. A GPU can perform enormous numbers of calculations, but if each step must fetch a growing cache before it can proceed, memory bandwidth becomes a hard constraint.

Each token feeds an attention step that reads a growing key-value cache in GPU memory.

Why adding hardware does not always fix it

It is tempting to think the answer is simply more GPUs. Sometimes it is. More hardware can serve more users at once, larger models, or longer contexts. But it does not automatically make one answer appear much faster, because the next-token dependency remains.

For a single request, splitting work across devices can add communication overhead. The system must move intermediate results between machines or chips, and that movement can consume the time gained by parallel computation. The smaller and more frequent the decoding steps, the more visible this overhead becomes.

For many simultaneous requests, batching helps. A batch groups decoding steps from different users so the hardware does a larger, more efficient operation. But batching introduces a trade-off: waiting briefly to form a good batch can improve total throughput while increasing latency for an individual user.

More hardware can increase capacity without improving single-user token speed.
Splitting one request across devices can be limited by communication between devices.
Batching improves hardware efficiency but can add waiting time.
Longer contexts increase cache size, which can reduce the number of requests that fit in memory.

⚠️

Capacity is not speed. A system can serve more requests per minute while each individual answer still streams at about the same rate.

More hardware serves many users, but one reply stays limited by the token dependency.

Common ways to relieve the bottleneck

There is no single cure for the decoding bottleneck. Useful systems combine several techniques, each with a cost. The right mix depends on whether the priority is lower latency, lower cost, higher quality, longer context, or more concurrent users.

Quantization stores or computes some model numbers with fewer bits. This can reduce memory use and data movement, often improving speed or capacity, but it may affect quality if applied too aggressively. Better kernels, which are low-level routines tuned for the hardware, can also reduce wasted work.

Speculative decoding uses a smaller or cheaper model to draft several possible tokens, then asks the larger model to verify them. When the drafts are accepted, the system can produce multiple tokens with less full-model work. When they are rejected, the benefit shrinks. Prompt trimming, response length limits, caching repeated prefixes, and choosing a smaller model can also be more effective than adding hardware.

Systems combine several techniques, each trading quality, cost, or complexity.

The decisions it changes

The decoding bottleneck is not only an engineering detail. It changes product, budget, and architecture decisions. A leader deciding between models should ask not only which model scores highest on an evaluation, but also how many output tokens it must produce, how quickly users need them, and how much concurrency the service must support.

A practitioner designing an application should treat output length as a first-class cost driver. A chat interface that writes long explanations, a code assistant that emits full files, and an agent that performs many small tool calls all stress decoding differently. The same model can feel fast in one workflow and slow in another.

This is also where service design matters. Streaming the answer can improve perceived responsiveness, even when total generation time is unchanged. Separating short interactive requests from long background jobs can protect user experience. Setting clear maximum output lengths can be more practical than scaling infrastructure indefinitely.

Choose model size based on required quality and required token speed, not quality alone.
Measure expected output length, not just prompt length.
Separate interactive workloads from batch or background workloads.
Set service-level objectives in user terms, such as time to first token and full answer time.
Design prompts and user interfaces to avoid unnecessary verbosity.

✅

Best lever. The cheapest token to decode is the one the product never needed to ask for.

How to measure it without fooling yourself

A useful measurement separates the phases of the request. Time to first token mostly reflects queueing, prompt processing, and scheduling. Inter-token latency measures the delay between streamed tokens during decoding. Total latency includes the whole journey from request to final output.

Throughput can also be measured in more than one way. Requests per second tells you how many jobs the service completes. Tokens per second tells you how much text it produces. These can move in different directions: a system may complete many short requests quickly while struggling with fewer long generations.

Good tests use realistic prompts, realistic output lengths, and realistic concurrency. A benchmark that asks for ten-token answers will hide problems that appear in a production assistant expected to write six hundred tokens. A single-user test will hide queueing and batching behavior that appears under load.

Key takeaways

Decoding is slow because generated tokens usually depend on earlier generated tokens.
Prefill and decoding stress hardware differently, so they need separate measurements.
The key-value cache saves recomputation but creates memory and bandwidth pressure.
More hardware can improve capacity without greatly improving one user's streaming speed.
Output length, batching, model size, and concurrency are architectural choices, not afterthoughts.
The most reliable optimizations are based on realistic prompts, outputs, and load.

Glossary

Token: A small piece of text, such as a word, part of a word, punctuation mark, or spacing fragment, that a language model reads or writes.
Decoding: The process of generating output tokens from a model's next-token predictions.
Prefill: The phase in which a model processes the input prompt before it begins generating output.
Key-value cache: Stored attention data that lets a transformer model reuse information from earlier tokens during generation.
Latency: The time a user or system waits for a response or part of a response.
Throughput: The amount of work a system completes in a given time, such as requests per second or tokens per second.
Batching: Grouping multiple requests or decoding steps so hardware can process them more efficiently.
Quantization: A technique that represents model numbers with fewer bits to reduce memory use and data movement.
Speculative decoding: A method in which a smaller or cheaper model drafts tokens that a larger model then verifies.