Continuous Batching & Paged Attention — LLM Inference Optimization & Serving

A large language model feels like a conversation, but a server sees a stream of uneven jobs: one person asks for a five-word answer, another pastes a long contract, and a third wants a page of code. Continuous batching and paged attention are two practical ideas that make this mess efficient. One keeps the graphics processing unit moving by mixing requests as they arrive and finish. The other stores each request’s growing memory in small reusable blocks instead of one fragile slab. Together they turn model serving from a waiting room into a managed traffic system.

Start with the token loop

A large language model (LLM) does not usually produce an answer all at once. It reads the prompt, then predicts the next token, then uses that token to predict the next one, and so on. A token is a small piece of text, often a word fragment, word, number, or punctuation mark.

This creates two phases. In the prefill phase, the model reads the input prompt and builds internal state for every token in that prompt. In the decode phase, it generates one new token at a time. Prefill is usually a large burst of work. Decode is a repeated sequence of smaller steps.

Serving many users is hard because every request has a different prompt length, output length, and stopping time. A short request may finish in one second while a long one keeps generating for a minute. If the server handles them one by one, expensive hardware waits too much. If it groups them too rigidly, fast requests get trapped behind slow ones.

A request reads the prompt, prefills internal state, then decodes one token per step until it stops.

Why simple batching is not enough

Batching means doing the same operation for several requests at once. A graphics processing unit (GPU) is built for this kind of parallel work, so batching is often the difference between a costly idle server and a useful one.

Traditional batching waits briefly, collects a group of requests, runs them together, and returns results when the group is done. That works well for fixed-size jobs, such as classifying images. Text generation is different because jobs do not end together. Each request keeps asking for one more token until it hits a stop condition, a token limit, or a user cancellation.

The awkward part is that a batch becomes ragged. Some rows still need more tokens, while others are finished. If the server waits for the whole batch to finish before accepting new work, capacity is wasted. If it starts too many separate batches, each batch may be too small to use the GPU well.

⚠️

The hidden queue. A model can look fast in isolation but feel slow in production if requests spend most of their time waiting to join an inefficient batch.

Continuous batching treats the batch as a living thing

Continuous batching, sometimes called dynamic batching for generation, changes the rule. Instead of forming a batch and keeping it fixed until every request is done, the server updates the batch at each decode step. Finished requests leave. New requests enter. Long-running requests continue without forcing everyone else to wait for them.

Think of an elevator that stops at every floor to let people in and out, rather than one that must empty completely before anyone new can board. The elevator is still shared, but membership changes continuously.

This is especially powerful during decode. Each active request needs one next token, so the server can combine many active requests into one GPU operation. After that step, it checks which requests stopped, streams tokens back to callers, admits waiting requests, and runs the next step.

It improves throughput by keeping the GPU supplied with work.
It can reduce time to first token for new requests because they need not wait for an old batch to drain.
It requires careful scheduling so large prompts, long outputs, and urgent requests do not crowd each other unfairly.

At each decode step the batch runs, finished requests leave, and waiting ones are admitted.

Attention needs memory that grows with the conversation

To generate the next token, a transformer model uses attention. Attention lets the model compare the current position with earlier tokens so it can use relevant context. Recomputing everything from the beginning for every new token would be wasteful, so servers save part of the attention state.

That saved state is commonly called the key-value (KV) cache. For each token already seen, the model stores key and value vectors that will be reused when predicting later tokens. The KV cache is not optional decoration; for long prompts and long answers, it can dominate serving memory.

The cache grows at different speeds for different requests. A request with a long prompt starts with a large cache. A request that generates many tokens grows over time. If the server cannot manage this memory well, it may run out of GPU memory even when the compute units are not fully busy.

📌

Compute is not the only limit. In LLM serving, memory layout can be as important as raw GPU speed.

The problem with contiguous cache allocation

A simple way to store the KV cache is to reserve one large continuous region of memory for each request, big enough for the maximum possible sequence length. This is easy to reason about, but it wastes memory whenever the request ends early or produces fewer tokens than expected.

Another simple method is to grow each request’s cache as needed in continuous memory. That can create fragmentation: free memory exists, but it is scattered in pieces that are not large enough for the next request. Fragmentation is familiar from storage systems and general-purpose memory allocators, but it is especially painful on GPUs because memory is scarce and expensive.

The core problem is uncertainty. The server often does not know in advance how long each answer will be. Reserving too much wastes capacity. Reserving too little forces copying, failure, or complex resizing.

Contiguous cache over-reserves and fragments; paged blocks grow on demand and are reused freely.

Paged attention borrows a trick from operating systems

Paged attention stores the KV cache in fixed-size blocks, often called pages, instead of requiring one continuous span per request. A request’s logical token sequence can be continuous, while its physical memory is spread across many blocks. A small table maps logical positions to the blocks that hold them.

This idea resembles virtual memory in an operating system. A program sees a neat address space, while the operating system maps that space to physical memory pages. In LLM serving, the model server applies a similar idea to attention memory: keep the sequence abstraction simple, but manage the underlying cache in reusable pieces.

When a request grows, the server attaches another block. When a request finishes, its blocks can be returned to a pool and reused by another request. This reduces waste from over-reservation and makes it easier to serve many requests with different lengths.

A page table maps a request's logical sequence to reusable blocks drawn from a shared pool.

Together, they make scheduling a systems problem

Continuous batching and paged attention solve different parts of the same serving problem. Continuous batching keeps compute busy by changing which requests are active at each step. Paged attention keeps memory usable by making each request’s growing cache easier to place, grow, share, and release.

The two ideas reinforce each other. Continuous batching creates frequent arrivals and departures inside the active set. Paged attention makes those changes cheaper because memory can be returned and reassigned in blocks. Without good memory management, a busy continuous batch can become blocked by cache pressure. Without good scheduling, efficient cache pages may still sit behind an underfilled GPU.

There are trade-offs. Page tables and block management add overhead. Small blocks reduce waste but can increase metadata work and memory indirection. Large blocks reduce overhead but may waste more tail space. A production server must choose policies that fit the model size, sequence lengths, latency targets, and hardware.

Batch policy decides which requests run together at each step.
Cache policy decides how much memory each request receives and when it is released.
Admission control decides whether a new request can start now or must wait to protect existing work.
Preemption policy decides whether a long or low-priority request can be paused so other work can proceed.

Decisions leaders and practitioners actually face

These techniques are not just implementation details. They shape cost, user experience, and reliability. A team choosing a model serving stack should ask how it handles mixed prompt lengths, streaming responses, cancellations, memory pressure, and priority traffic. Benchmarking only one prompt size or one output length can hide the very behavior that matters in production.

Practitioners should measure both throughput and latency. Throughput says how many tokens or requests the system completes over time. Latency says what one user experiences, including time to first token and time to complete the answer. Continuous batching can improve total capacity while still requiring careful tuning to avoid starving small or urgent requests.

Leaders should also ask what failure looks like. Does the service reject new requests cleanly when the cache is full, or does it slow everything down? Can it isolate tenants, reserve capacity for important workloads, and report why requests are waiting? The best system is not merely fast on a dashboard; it is predictable under uneven, real traffic.

✅

Useful benchmark. Test with a realistic mix of short chats, long prompts, long generations, cancellations, and bursts, not only with one tidy average request.

Key takeaways

LLM serving has two phases: prefill reads the prompt, and decode generates one token at a time.
Continuous batching updates the active batch after each decode step so finished requests leave and new requests enter.
The KV cache stores reusable attention state, and it often becomes a major GPU memory constraint.
Paged attention stores the KV cache in reusable blocks, reducing waste and fragmentation from uneven sequence lengths.
The best serving design balances throughput, latency, memory pressure, fairness, and predictable overload behavior.

Glossary

Token: A small unit of text, such as a word fragment, word, number, or punctuation mark, that a language model reads or generates.
Large language model (LLM): A model trained on large amounts of text to predict and generate token sequences.
Graphics processing unit (GPU): A processor designed to run many similar calculations in parallel, commonly used for machine learning workloads.
Prefill: The serving phase in which the model reads the input prompt and builds internal state for its tokens.
Decode: The serving phase in which the model generates new tokens, usually one step at a time.
Continuous batching: A scheduling method that updates a generation batch as requests arrive, finish, or cancel instead of keeping the batch fixed.
Key-value (KV) cache: Stored attention state from earlier tokens that the model reuses when generating later tokens.
Paged attention: A memory management technique that stores the KV cache in fixed-size blocks and maps logical token positions to those blocks.
Fragmentation: A memory condition where free space exists but is split into pieces that are hard to use efficiently.