Isolation and Mitigations: Cache Partitioning, Quotas, and the Throughput Cost

Every attack in this series reduces to the same sentence: state shared across a trust boundary leaks through timing or cost. The prefix cache leaks because it is shared; the decoding timing leaks because the fast path is shared; the resource pool is exhausted because capacity is shared. This uniformity is good news for the defender, because it means the fixes are uniform too. There are exactly three things you can do with a shared channel — eliminate it, mask it, or meter it — and the art of hardening a serving stack is choosing which to apply where, knowing that each one costs some of the efficiency the sharing was there to provide. Isolation is not free, so the goal is not maximum isolation but the right isolation, chosen deliberately per tenant.

Three strategies for every channel

Confronted with a shared-state side channel, a defender has three moves. Eliminate: remove the sharing across the trust boundary, so the channel does not exist — partition the cache per tenant, isolate scheduling, give each tenant dedicated capacity. Mask: keep the sharing but make the observable independent of the secret — pad timing to fixed quanta, pace token emission, add noise. Meter: keep the sharing and the signal but cap how much an attacker can observe — per-tenant rate limits and quotas that bound total extractable information or induced cost.

These are not interchangeable; they have different strengths. Elimination is the strongest — a channel that does not exist cannot leak — but the most expensive, since it forgoes the efficiency of sharing entirely. Masking preserves some sharing but is often imperfect (noise can be averaged away) and must be paired with metering to be robust. Metering alone caps the damage but does not close the channel, so it is a backstop, not a primary fix. The strongest posture combines them: eliminate where sensitivity is high, mask where sharing must remain, and meter everywhere as a floor.

The reason to name the three explicitly is that it turns an open-ended security problem into a routing decision: for each shared optimization, pick eliminate, mask, or meter based on the sensitivity of the traffic and the cost of the fix. The rest of this article applies that routing.

Cache partitioning

The prefix-cache membership oracle is closed by elimination: partition caches per tenant so a hit can only ever reflect the same tenant's own prior requests. With per-tenant cache namespaces, an attacker probing for another tenant's prefix always misses, and the oracle carries zero cross-tenant information. This is the cleanest possible fix — it drives the channel's capacity to zero — and it is the default a confidentiality-sensitive deployment should reach for.

The cost is cache efficiency: cross-tenant prefix sharing (where many tenants using the same public prefix share one cached copy) is forgone, so total cache memory usage rises and the hit rate for shared public prefixes falls. For deployments where most cacheable prefixes are tenant-specific anyway, this cost is small; for deployments dominated by a common shared prefix, it is larger. The middle path is to share only explicitly public, non-sensitive prefixes and partition everything else, capturing most of the efficiency while closing the sensitive channel.

Where full partitioning is too costly, masking (adding timing noise to hit/miss) and metering (rate-limiting probes) are fallbacks, but they only slow an attacker rather than closing the channel. Partitioning is the control that actually eliminates the oracle, and it should be the target for any prefix that could carry sensitive content.

Constant-time-ish serving

The decoding-timing and content-dependent-latency channels are closed by masking: make the observable timing independent of the secret content. The practical form is token pacing — buffer generated tokens and emit them at a fixed interval so the client sees a uniform stream regardless of how fast the tokens were actually produced. This masks speculative-decoding acceptance and other content-dependent speedups from the emission channel, closing the timing leak for observers of the stream.

True constant-time execution is expensive and often impractical for large models, so serving aims for constant-time-ish: the internally variable computation is allowed, but the externally observable timing is normalized. Pacing achieves this for the output stream; padding request latency to fixed quanta achieves it for end-to-end timing. The cost is latency — a paced stream is no faster than its fixed interval even when the model could go faster — so the speedup is preserved internally for capacity but not passed through to the user's observable timing.

As with all masking, it must be paired with metering. Pacing removes the per-query timing signal, but a determined attacker facing residual noise averages over many queries, so per-tenant rate limits cap the total observations and therefore the total leak. Masking plus metering together bound the channel; masking alone leaves an averaging attack open.

Quotas and metering

Metering is the universal backstop and the primary defense against resource amplification. Per-tenant quotas — tokens, cost, and requests per time window — cap aggregate consumption, so denial-of-wallet and sponge attacks cannot exceed a budget no matter how clever a single request is. Per-request bounds — max input and output tokens, compute and wall-clock budgets, memory limits — cap what any one request can consume, so amplification along each dimension is bounded. Together they bound total cost as bounded-amplification times bounded-actions.

Metering also backstops the confidentiality channels. Even after partitioning and pacing, some residual leak may remain; a per-tenant probe rate limit caps the total queries an attacker can issue and therefore the total bits they can extract, keeping residual-leak-times-query-rate under a chosen budget. This is why metering appears in every mitigation: it does not close a channel, but it bounds any channel's total exploitation, which is exactly what an averaging or amplifying attacker needs unbounded.

The cost of metering is the least of the three strategies — it mostly affects abusive traffic, not legitimate use — which is why it should be applied everywhere as a floor. The design point is to set quotas generously enough for real workloads and tightly enough that the worst-case extractable information or induced cost is acceptable, and to alert on anomalies so a tenant approaching their cap is noticed quickly.

Choosing the trade per tenant

Because every fix costs throughput, the controls should be allocated by tenant sensitivity rather than applied uniformly. High-sensitivity tenants — those whose prompts carry credentials, private data, or proprietary templates — get the strong combination: partitioned caches, paced timing, isolated scheduling, and tight quotas. Low-sensitivity, public-workload tenants can enjoy full sharing and the associated speed, with only metering as a floor. The decision below routes a tenant to a posture based on the sensitivity of their traffic and the platform's cost tolerance.

The guiding principle is that maximum throughput and maximum confidentiality are mutually exclusive for shared serving, so the platform must choose, and the safe default for anything sensitive is isolation. The failure mode to avoid is a performance default (full sharing enabled for everyone because it is fast) silently exposing a sensitive tenant who assumed isolation. Making the posture an explicit, per-tenant decision — and defaulting sensitive traffic to isolation — prevents that.

The synthesis is that inference-serving security is a deliberate isolation-versus-efficiency allocation. Eliminate the sharing for sensitive tenants, mask what must remain shared, meter everyone, and measure the residual leak so the chosen throughput cost buys a known confidentiality guarantee. The mechanics from Kwon, Leviathan, and Shumailov and colleagues make serving fast; the discipline here is to spend a measured amount of that speed on isolation wherever the traffic deserves it.

A deployment blueprint

Put together, a hardened multi-tenant serving stack looks like this. Caches are partitioned per tenant, with cross-tenant sharing limited to explicitly public prefixes. Sensitive traffic is served with paced token emission and isolated scheduling so timing does not depend on content or co-tenants. Every tenant has token, cost, and request quotas, and every request has cost bounds, with anomaly alerting on both confidentiality-probe rates and cost. The posture for each tenant is an explicit configuration, defaulting sensitive traffic to isolation, and the residual leak for each shared path has a measured bits-per-query figure kept under budget.

The overarching lesson is that shared-serving side channels are not exotic bugs but the direct, predictable consequence of the optimizations that make serving economical, so they are managed rather than eliminated: you decide, per tenant, how much sharing to allow, close or mask what the sensitive tenants cannot afford to share, meter everyone, and measure what remains. Inference serving will always trade isolation for efficiency; the security work is to make that trade consciously, per tenant, against a measured leakage budget — never to discover in production which optimization you shared too widely.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

serving_posture_policy:
  default_for_sensitive: full_isolation
  postures:
    full_isolation:
      cache: per_tenant
      timing: paced
      scheduling: isolated
      quotas: tight
    share_and_meter:
      cache: public_prefixes_only
      timing: exposed
      quotas: enforced
  measurement:
    residual_bits_per_query: tracked
    leakage_budget: defined
  alerting:
    probe_rate: on
    cost_anomaly: on

# DEFENSIVE REGRESSION — did our mitigations close the channels?
function residual_leak(service):
    cache = membership_leak(service, canary_prefix)   # should be ~0 if partitioned
    timing = timing_content_corr(service, canary_prompts) # ~0 if paced
    cost = amplification_probe(service, benign_stress)    # <= budget if bounded
    assert cache < EPS and timing < EPS and cost <= AMP_BUDGET, "leak remains"
    # Run in CI so a config change cannot silently reopen a channel.

Glossary

Eliminate / mask / meter

The three strategies for a shared-state side channel: remove the sharing, make the signal secret-independent, or cap total observation.

Cache partitioning

Isolating caches per tenant so a hit reflects only that tenant's own prior requests, closing the membership oracle.

Constant-time-ish serving

Normalizing observable timing to be independent of content, even though internal computation varies.

Token pacing

Emitting generated tokens at a fixed interval to mask content-dependent generation speed.

Per-tenant quota

A cap on a tenant's tokens, cost, or requests per window, bounding aggregate abuse and residual leakage.

Per-request bound

A cap on a single request's tokens, compute, memory, or time, bounding amplification per dimension.

Residual leak

The information a channel still carries after mitigation, tracked as bits-per-query against a budget.

Isolation-efficiency trade

The unavoidable tension in shared serving between confidentiality (isolation) and throughput (sharing).

References

1. Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (arXiv 2309.06180)
2. Leviathan et al., Fast Inference from Transformers via Speculative Decoding (arXiv 2211.17192)
3. Shumailov et al., Sponge Examples: Energy-Latency Attacks on Neural Networks (arXiv 2006.03463)
4. Kocher, Timing Attacks on Implementations of Diffie-Hellman, RSA, DSS, and Other Systems (CRYPTO 1996)
5. NIST AI 100-2 e2023, Adversarial Machine Learning: A Taxonomy and Terminology