Prefix Caching and Reuse — Speculative Decoding & KV-Cache

A large language model does not read a prompt the way a person reads a page; it turns the text into tokens, performs a large amount of math over those tokens, and then begins generating an answer. Prefix caching is a simple idea built on that fact: when many requests start with the same text, the system can keep the expensive early work and reuse it. Done well, it lowers latency and cost without changing the answer. Done carelessly, it can hide bugs, leak assumptions across users, or save very little.

Start with the prefix

A prefix is the beginning of a request. In a chat application, it might include the system instruction, safety rules, tool descriptions, output format, examples, and a long reference document. The user’s newest question often comes after all of that.

Prefix caching works when that beginning is identical across requests. If request A and request B start with the same first 8,000 tokens, the model-serving system can reuse work from request A when handling request B, then continue from the point where the requests differ.

This is different from caching a finished answer. Answer caching says, “I have seen this whole question before.” Prefix caching says, “I have seen the start of this computation before.” That distinction matters because the user’s final question can be new while the expensive setup remains the same.

📌

Exact means exact. Most prefix caches rely on the same token sequence, so a small change in spacing, ordering, or hidden metadata can prevent reuse.

Prefix caching reuses the start of the computation while answer caching reuses whole replies.

Why the beginning costs so much

A model request has two broad phases. First is prefill: the model processes the input tokens you supplied. Second is generation: the model produces new output tokens one at a time. Prefix caching mainly reduces the prefill phase.

Prefill can be expensive because every input token influences later tokens through attention, the mechanism that lets the model relate words and fragments across the context. Long prompts, many examples, large tool schemas, and retrieved documents all increase the amount of prefill work.

The user notices this cost as time to first token, which is the delay before the first generated token appears. Operators notice it as consumption of graphics processing unit resources, often called GPU resources, because GPUs perform the parallel math used for model inference.

What is actually reused

During prefill, the model builds internal state for the tokens it has processed. In transformer models, an important part of that state is commonly called the key-value cache, or KV cache. It stores mathematical representations that the model would otherwise need to recompute.

A prefix cache keeps KV cache entries for a shared beginning of a prompt. When a later request starts with the same tokens, the serving system can attach the saved state and continue processing only the remaining tokens.

The cache does not contain a human-readable summary of the prompt, and it is not a database of facts. It is intermediate model state. That is why it is fast, but also why it must be treated as sensitive: it is derived from user or business content even when it is not easily readable.

Design prompts for reuse

Prefix caching rewards stable structure. Put the most commonly repeated material first: the system message, policy text, tool definitions, response schema, and few-shot examples. Put per-user and per-request material later, where it will not break the shared prefix.

A common mistake is to place volatile values near the start of the prompt: timestamps, request identifiers, user names, trace headers, randomly ordered tools, or retrieved passages that change on every call. Any of these can turn a reusable prefix into a new prefix.

Prompt templates should be treated like application interfaces. If a team changes the order of fields or reformats a schema on every deployment, cache hit rates may fall even when the model behavior appears unchanged.

Keep the shared system instruction byte-for-byte stable when possible.
Sort tool definitions and schema fields consistently.
Move timestamps, request identifiers, and user-specific facts after the reusable block.
Avoid adding invisible variation such as random whitespace or nondeterministic serialization.
Version prompt templates deliberately so cache changes are expected, not accidental.

Stable material goes first and volatile per-request content goes last to keep prefixes shared.

Correctness and safety boundaries

Prefix caching should not change the model’s semantics. A request with a reused prefix should be equivalent to the same request computed from scratch. If reuse changes the answer, the cache boundary is wrong or the serving system has a bug.

The main safety issue is isolation. A cache entry derived from one tenant, user, security label, model version, or tool configuration must not be reused where it does not belong. The cache key should include every factor that changes meaning or access rights, not merely the visible text.

This is especially important in enterprise systems that use identity and authorization, such as applications integrated with Microsoft Entra ID. If two users see different documents or tools, their prefixes may look similar but have different permissions behind them.

⚠️

Do not share trust. Never let a prefix cache cross tenant, permission, model, or policy boundaries unless the platform explicitly enforces safe isolation.

The cache key must include every factor that changes meaning or access rights.

Measure the value, not the feature

The useful question is not whether prefix caching is enabled. The useful question is whether it improves the workload you actually run. A chat system with a stable 12,000-token setup can benefit greatly. A system where every request starts with different retrieved content may see little gain.

Measure cache hit rate, saved input tokens, time to first token, end-to-end latency, throughput, and cost per successful task. Also watch memory pressure, because cached KV state consumes capacity that could otherwise serve new requests.

Results are workload-specific. Longer shared prefixes usually create more upside, but only if requests arrive often enough for the cached state to still be available and if the cache is not evicted before reuse.

The upside grows with longer shared prefixes and more frequent requests.

The real decisions

For a practitioner, prefix caching affects prompt layout, routing, observability, and deployment discipline. For a leader, it affects capacity planning and the economics of long-context applications. The feature is technical, but the decision is architectural: are you building many requests around a stable shared context, or are you creating a new context every time?

In platforms such as Microsoft Foundry, teams should evaluate prefix reuse alongside model choice, context length, retrieval strategy, tool design, and data isolation. The cheapest token is often the one you do not ask the model to process again, but only if reuse remains correct and secure.

A sensible rollout starts with one high-volume path, a known prompt template, and clear before-and-after measurements. If the results are strong, standardize the prompt patterns that made reuse possible.

Choose workloads with long, stable prefixes and enough repeated traffic.
Set service-level objectives, or SLOs, for first-token latency and full-response latency.
Decide which boundaries must be part of the cache key, including tenant, user permissions, model version, and tool set.
Budget memory for cached state, not just compute for live inference.
Treat prompt template changes like code changes, with review and regression measurement.

✅

Best fit. Prefix caching is strongest when many requests share a large, stable beginning and differ only near the end.

Key takeaways

Prefix caching reuses the computed state for the identical beginning of a model request.
It mainly reduces prefill work, which can improve time to first token and lower compute cost.
Stable prompt structure matters: put shared instructions, schemas, and examples before volatile request data.
The cache must respect tenant, permission, model, tool, and policy boundaries.
Measure cache hit rate, saved tokens, latency, cost, and memory pressure before declaring success.
Prefix caching is not answer caching; it can help even when each user asks a different final question.

Glossary

Prefix: The beginning portion of a model request that may be shared by many requests.
Token: A chunk of text, such as a word, part of a word, or symbol, that a model processes as an input or output unit.
Prefill: The phase in which a model processes the input tokens before it begins generating new tokens.
Key-value cache: Stored internal model state, often shortened to KV cache, that lets a transformer model avoid recomputing attention information for earlier tokens.
Cache hit: A request finds a matching cached entry and can reuse it instead of recomputing the same work.
Time to first token: The delay between sending a request and receiving the first generated output token.
Context window: The maximum amount of input and output tokens a model can handle in one request.
Tenant isolation: A security boundary that keeps one customer’s or organization’s data and computed state separate from another’s.
Service-level objective: A measurable reliability or performance target, often shortened to SLO, that a service aims to meet.