Speculative and Parallel Decoding — Speculative Decoding & KV-Cache

A language model feels slow for a simple reason: it usually writes one small piece of text, called a token, then uses that token to decide the next one. That step-by-step habit is useful, because each new word depends on the words before it, but it also creates a long queue of tiny decisions. Speculative and parallel decoding are ways to shorten that queue. They do not make the model smarter. They make the act of reading out its answer more efficient.

Start with the bottleneck

A modern language model turns text into tokens, which are short chunks such as words, parts of words, punctuation marks, or spaces. To answer a prompt, the model repeatedly predicts a probability distribution over the next possible token, chooses one token, appends it to the text, and starts again.

This repeated process is called decoding. The common form is autoregressive decoding, which means the model generates each token using the tokens that came before it. The dependency is strict: token 50 cannot be chosen until token 49 exists.

The expensive part is a forward pass, which is one run through the model to compute the next-token probabilities. Hardware such as graphics processing units (GPUs) is excellent at doing large amounts of math in parallel, but plain autoregressive decoding asks it to perform many small sequential steps. That mismatch is the core bottleneck.

What parallelism can and cannot change

Parallel decoding is an umbrella idea: use the hardware to evaluate more than one useful possibility at the same time. It can mean checking several proposed tokens in one model run, exploring several branches of a response, or serving many users together in a batch.

Parallelism cannot remove the logic of language. If the next token truly depends on the previous token, the system must respect that dependency somehow. The trick is to find work that can be done before a decision is final, or work that can be checked together after a guess is made.

This distinction matters because faster decoding is not the same as a smaller model or a better answer. The model weights, training data, and reasoning ability are unchanged. The gain comes from reducing idle time, reducing the number of expensive target-model steps, or filling each step with more useful computation.

📌

Speed is not intelligence. Decoding techniques change how answers are produced, not what the model has learned.

The speculative idea

Speculative decoding begins with a guess. A smaller or cheaper draft model predicts several future tokens quickly. Then the larger target model, which is the model whose behavior you want, checks those draft tokens in parallel.

The target model does not have to run once per draft token. Because transformer models can compute next-token probabilities for every position in a proposed sequence during one forward pass, the target model can verify a whole short run of draft text at once.

If the draft model guessed tokens the target model would also accept, the system keeps them and jumps forward several tokens. If the draft diverges, the system keeps the longest acceptable prefix, corrects the first bad point using the target model's probabilities, and continues. When implemented with the right sampling correction, speculative decoding can preserve the same output distribution as sampling directly from the target model.

A cheap draft model guesses tokens the target model verifies in one pass and accepts.

Why the guess can be safe

The word speculative can sound risky, but the risk is controlled by verification. The target model remains the authority. The draft model only proposes. It is like a junior editor writing a sentence and a senior editor approving the parts that match their own judgment.

For deterministic decoding, such as always choosing the most likely token, the check can be simple: keep draft tokens that match the target model's chosen tokens. For probabilistic sampling, the system must be more careful. It compares the draft model's probability for a token with the target model's probability and uses an acceptance rule that avoids biasing the final distribution.

The best speedups come when the draft model is both fast and aligned with the target model. If it is too weak, many guesses are rejected and the extra draft work is wasted. If it is too large, it may predict well but cost too much to be useful.

✅

Exact when corrected. Speculative decoding can be lossless for sampling if the acceptance and correction rules preserve the target model's distribution.

The target model is the authority, keeping matches and correcting the first divergence.

Other forms of parallel decoding

Speculative decoding is the best-known pattern, but it is not the only way to parallelize generation. Some systems generate a tree of possible continuations and ask the target model to verify multiple branches. This can find more acceptable tokens per target-model pass, especially when the first guess has several plausible alternatives.

Some model architectures add extra prediction heads that try to forecast token two, token three, or token four while the normal head predicts token one. These heads can make the model propose its own future tokens, reducing the need for a separate draft model. The model still needs a way to verify or accept those proposals.

Batching is another kind of parallelism. Instead of making one user's request faster by itself, the serving system groups work from many users so each GPU pass is fuller. Batching improves throughput, which is the number of tokens served per second across all users, but it may add waiting time for an individual request if managed poorly.

Non-autoregressive generation tries to produce many tokens at once and refine them. It can be powerful in tasks with predictable structure, such as some translation or editing settings, but open-ended chat is harder because each new token can change the best continuation.

Speculative verification: a draft proposes, the target model approves.
Tree verification: several possible continuations are checked together.
Multi-token heads: one model predicts near-future tokens directly.
Batching: many users' decoding steps are grouped for hardware efficiency.
Non-autoregressive methods: many tokens are produced and refined together.

Several patterns fill each hardware pass with more useful decoding work.

The systems view

In production, decoding speed depends on more than the algorithm. The system must manage memory, queues, network calls, and the key-value cache, often shortened to KV cache. The KV cache stores internal attention data from earlier tokens so the model does not recompute the whole conversation at every step.

Speculative decoding changes cache behavior. Accepted draft tokens can be added to the target model's cache, but rejected tokens must not pollute it. Tree-based methods can create several temporary cache branches, which may improve speed but increase memory pressure.

There is also a scheduling question. A server may be handling short chats, long summaries, tool-using agents, and streaming responses at the same time. The best decoding strategy for one request may be wasteful for another. Long, predictable outputs often benefit more from speculation than short, surprising answers.

Accepted tokens enter the target cache while rejected ones must never pollute it.

Decisions for leaders and practitioners

The practical question is not whether speculative or parallel decoding is clever. It is whether it improves the service you are accountable for. Start with the user-facing goal: lower first-token latency, faster full responses, lower cost per token, higher throughput, or steadier performance under load.

Measure before and after with realistic prompts. Include short and long requests, easy and difficult domains, streaming and non-streaming responses, and any tool calls or retrieval steps. A benchmark made only of long, predictable completions can make speculation look better than it will feel in a mixed product.

Watch the full cost equation. A draft model uses compute. Extra verification paths use memory. Larger batches may raise throughput but hurt latency. A technique that saves target-model passes can still be a bad fit if it increases tail latency, which is the slowest slice of user experiences, or misses a service-level objective (SLO), which is a promised performance target.

Finally, decide how much exactness you require. In many serving stacks, a lossless speculative method is preferable because it preserves the target model's behavior. Approximate methods may be acceptable for constrained tasks, but they should be treated as product changes, not invisible optimizations.

⚠️

Benchmark honestly. Use your real traffic mix, because decoding speedups vary sharply with prompt type, output length, and load.

A simple mental model

Think of ordinary decoding as walking one stepping stone at a time. Speculative decoding sends a scout across the next few stones, then asks the main traveler to confirm which stones are safe. Parallel decoding more broadly looks for every place where the traveler, scout, and road crew can work at the same time without changing the destination.

The important discipline is to keep authority clear. The draft can suggest, the hardware can check in parallel, and the serving layer can schedule work efficiently, but the target model's probability distribution is the thing users experience. Good systems make decoding faster while keeping that contract visible.

Key takeaways

Autoregressive decoding is slow because each token depends on the tokens before it.
Speculative decoding uses a cheap draft model to propose tokens and a target model to verify them in parallel.
With the right acceptance rule, speculative sampling can preserve the target model's output distribution.
Speedups depend on draft quality, output length, hardware utilization, cache handling, and traffic mix.
Parallel decoding includes speculation, tree verification, multi-token heads, batching, and non-autoregressive methods.
Leaders should evaluate decoding changes against latency, throughput, cost, exactness, and real user workloads.

Glossary

Token: A small chunk of text, such as a word, part of a word, punctuation mark, or space, that a language model reads and writes.
Decoding: The process of turning a model's next-token probabilities into an actual sequence of output tokens.
Autoregressive decoding: A generation method in which each new token is produced using the tokens that came before it.
Forward pass: One run through a model that computes outputs such as next-token probability distributions.
Draft model: A smaller or cheaper model used to propose future tokens before the target model checks them.
Target model: The authoritative model whose behavior and probability distribution the serving system aims to preserve.
KV cache: Stored key-value attention data from earlier tokens that lets a transformer model avoid recomputing the whole context at every decoding step.
Latency: The time a user waits for a response or for a specific part of a response, such as the first token.
Throughput: The amount of work a system completes over time, often measured as tokens served per second.