KV-Cache Management & Paging — Speculative Decoding & KV-Cache

A large language model does not answer in one motion. It reads the prompt, then writes one token at a time, where a token is a small piece of text such as a word fragment, word, or punctuation mark. Without a memory of what it has already read and written, the model would have to redo the same work at every step. The key-value cache, usually shortened to KV cache, is the engineering trick that prevents that waste. It is also one of the main reasons modern inference systems are hard to run well.

Start with the work the model repeats

A transformer is a model design that processes tokens by comparing each token with other tokens in the same sequence. The comparison mechanism is called attention. Attention lets the model decide that, for example, the word it is about to write should depend more on a name from three paragraphs ago than on a nearby comma.

When the model receives a prompt, it turns each token into numbers and passes those numbers through many layers. In each layer, attention creates three related sets of numbers for each token: queries, keys, and values. A query is what the current token is looking for. A key is what a past token offers for matching. A value is the information retrieved when there is a useful match.

During generation, the model adds one new token at a time. The keys and values for earlier tokens do not change. Recomputing them again and again would waste time and energy. So the system stores them. That stored state is the KV cache.

What the KV cache actually contains

The KV cache is not a copy of the prompt text. It is a set of numeric tensors, where a tensor is a shaped array of numbers. For every active request, for every generated or prompt token that must remain in context, and for many layers of the model, the cache stores key vectors and value vectors.

This matters because the cache grows with the length of the conversation. A short prompt may need little cache memory. A long document, a multi-turn chat, or many parallel users can fill memory quickly. The model weights are mostly fixed once the model is loaded, but the KV cache expands and shrinks as traffic changes.

The cache is also request-specific. Two users asking different questions cannot usually share the same cache, because their token histories differ. Some systems can share common prefixes, such as a repeated system instruction, but once the histories diverge the cache must be tracked separately.

📌

Not text memory. The KV cache stores internal attention state, not readable words or a database of facts.

Why memory, not math, often becomes the limit

Inference has two main phases. Prefill is the phase where the system reads the prompt and builds the initial cache. Decode is the phase where it produces new tokens one at a time and appends new cache entries. Prefill is often heavy on computation because many prompt tokens can be processed together. Decode is often constrained by memory movement because each new token must look back across the cached context.

A graphics processing unit, or GPU, is designed to do many numeric operations in parallel, but its memory is finite. The KV cache competes for that memory with model weights, temporary working space, and other requests. The longer the context window and the larger the batch of active requests, the more cache memory is needed.

This is why a system can have plenty of arithmetic capacity and still slow down or reject work. The bottleneck is not simply whether the model can think fast. It is whether the serving system can place, find, move, and retire the right cache blocks at the right time.

⚠️

Long context has a bill. A larger context window increases capability, but it also increases memory pressure for every active sequence that uses it.

Prefill is compute-heavy building the cache while decode is memory-bound reading it.

The simple but wasteful approach

The most direct way to store a cache is to reserve one contiguous stretch of memory for each request. Contiguous means the cache sits in one unbroken region. This is easy to reason about and fast to address, but it becomes inefficient when requests are different lengths.

Imagine a system that reserves space for a maximum conversation length, even when many users ask short questions. Much of that reserved space may sit unused. If the system instead grows each request as needed, it can run into fragmentation, where free memory exists but is split into pieces that are hard to use for a growing sequence.

The serving problem is therefore not just how much memory exists in total. It is how flexibly that memory can be assigned while hundreds or thousands of sequences start, grow, finish, and get cancelled.

Paging borrows an old memory idea

Paging is a memory management technique that divides storage into fixed-size chunks called pages or blocks. Instead of requiring one long contiguous region for a request, the system gives the request a list of blocks. The blocks can live in different places, and a small mapping table records their order.

This is similar in spirit to virtual memory in an operating system, but adapted for model inference. A sequence might think it has token positions 1 through 10,000 in a neat line. Under the hood, those positions can be stored across many separate cache blocks. The attention kernel, which is the low-level program that performs attention efficiently, uses the mapping to read the right blocks.

Paged KV cache management improves utilization because memory can be allocated in smaller increments and returned to the pool as soon as a sequence ends. It also makes continuous batching easier, because new requests can join and old requests can leave without forcing the whole batch into a rigid layout.

A growing sequence gets fixed-size blocks tracked by a mapping table the kernel reads.

Scheduling is cache management in motion

Paging solves the placement problem, but it does not decide which work should run next. That job belongs to the scheduler. The scheduler chooses which requests are in the active batch, how much prefill work to admit, and how many decode steps to run before reconsidering the queue.

Good scheduling balances latency, throughput, and memory risk. Latency is how long one user waits. Throughput is how much total work the system completes over time. Memory risk is the chance that an admitted batch grows beyond available cache space. These goals pull against one another.

Modern serving systems often use continuous batching. Instead of waiting for a whole batch to finish, the system adds and removes requests at token boundaries. Paging makes this practical because each request can carry its own block map while sharing the same execution step with other requests.

The scheduler admits, decodes, frees, and rechecks work at every token boundary.

The decisions teams actually face

KV-cache policy turns abstract model capability into concrete product behavior. A leader may ask for longer context, lower cost, more concurrent users, and faster responses. The serving team has to translate those goals into limits and trade-offs that the hardware can sustain.

The key decision is not whether paging is good. In high-utilization systems, it usually is. The real questions are about block size, admission control, maximum context length, cache sharing, eviction behavior, and the service-level objective, or SLO, that defines acceptable latency and reliability.

Practitioners should test these choices with the traffic shape they actually expect. A benchmark full of short prompts will hide problems that appear with long documents. A benchmark with one user will miss the memory pressure created by many users decoding at once.

Set a maximum context length that matches the product need, not just the model maximum.
Choose admission rules that reserve memory for requests already in progress.
Measure time to first token separately from time per output token.
Track cache utilization, block fragmentation, and request cancellations.
Decide whether common prefixes, such as system instructions, are worth sharing safely.

Failure modes and practical safeguards

The most obvious failure mode is running out of memory. A safer system rejects or queues new work before it threatens active requests. A less safe system admits too much, then stalls, crashes, or kills requests after users have already waited.

Fragmentation can still matter with paging, especially if block sizes are poorly chosen. Small blocks reduce waste but add mapping overhead. Large blocks make addressing simpler but waste more space at the end of sequences. There is no universal best size; it depends on model shape, attention kernels, hardware, and traffic.

Cache correctness is also a safety issue. Blocks must not leak across users, must be cleared or retired correctly, and must map to the right sequence positions. A stale or misassigned cache block can produce wrong output in ways that are difficult to diagnose because the error is inside numeric state, not application text.

⚠️

Cache isolation matters. A KV cache is internal state for a specific request, so block reuse must never expose one user's state to another.

Key takeaways

The KV cache stores reusable attention keys and values so the model does not recompute past context at every generated token.
Cache memory grows with active requests, context length, model size, and generated output length.
Paging stores cache state in fixed-size blocks, reducing waste and making dynamic batching easier.
The scheduler and the cache manager must work together because every admission decision consumes future memory.
Long-context capability is a product and infrastructure trade-off, not just a model feature.
Safe operations require monitoring utilization, fragmentation, latency, cancellations, and cache isolation.

A safe server queues new work before an admitted batch threatens active requests.

Glossary

Token: A small piece of text, such as a word fragment, word, or punctuation mark, that a language model reads or writes.
Transformer: A neural network design that uses attention layers to relate tokens to other tokens in the same sequence.
Attention: The mechanism that lets a model weigh which earlier tokens matter most when processing or generating a token.
KV cache: Stored key and value tensors from attention layers that let inference reuse past context instead of recomputing it.
Prefill: The inference phase that reads the input prompt and builds the initial KV cache.
Decode: The inference phase that generates output tokens one at a time while extending and reusing the KV cache.
Paging: A memory management method that stores data in fixed-size blocks and uses a mapping table to present them as an ordered sequence.
Continuous batching: A serving technique that adds and removes requests at token boundaries instead of waiting for an entire batch to finish.
Service-level objective: A measurable target for service behavior, such as a latency or reliability goal that the system is designed to meet.