Caching, Routing & Model Tiering — Cost & Capacity Engineering

Every AI request spends three things: time, money, and trust. Caching reduces repeat work, routing sends each request to the place most likely to handle it well, and model tiering avoids using a large, expensive model when a smaller one is enough. Together they turn a single-model experiment into an engineered system with predictable behavior.

Start with the unit of work

A model is software that has learned patterns from data and can produce a prediction, answer, classification, image, or other output. When an application asks a model to do something, that request is often called an inference request. The system sends input, waits for the model to run, and receives output.

From first principles, an inference request is like any other remote computation. It may be slow, it may fail, it may cost money each time, and it may return a result that needs checking. If thousands of users ask similar questions, doing the full computation every time can waste capacity and introduce avoidable delay.

Caching, routing, and tiering are three controls placed around the model. A cache asks, Have we already done this safely? A router asks, Where should this go? A tiering policy asks, How powerful a model does this task really need?

Caching, routing, and tiering are three controls placed around the model to save work and match effort.

Caching means remembering with rules

A cache is a short path to a previous result. Instead of recomputing an answer, the system looks up whether it has already seen an equivalent request and whether the old answer is still valid. If yes, it returns the stored result. If no, it calls the model and may store the new result for later.

The hard part is not storing data. The hard part is deciding what counts as the same request. In a normal web application, the cache key might be a uniform resource locator, or URL. In an AI application, the input may include a prompt, system instructions, selected tools, user identity, retrieved documents, model version, temperature setting, and safety policy. Change one of these and the answer may change.

There are two broad styles. Exact caching reuses an answer only when the key matches exactly. Semantic caching reuses an answer when a new request is close enough in meaning to an old one. Exact caching is safer and simpler. Semantic caching can save more work, but it needs tighter rules because similar wording does not always mean the same intent.

Cache stable, low-risk work first, such as document summaries, product descriptions, classifications, and repeated internal help questions.
Include the model version, prompt template, retrieval sources, and relevant permissions in the cache key.
Set a time to live so old answers expire when facts, policies, or source documents may have changed.
Do not cache private or regulated outputs unless the storage, access controls, and retention policy are explicitly designed for that use.

⚠️

Similarity is not permission. A semantically similar request may still belong to a different user, tenant, document set, or legal context.

Exact caching reuses an answer only on an identical key; semantic caching reuses when meaning is close enough.

Routing chooses the best path for each request

A router is a decision point. It receives a request, reads enough context to understand what kind of work is being asked for, and sends the request to a model, tool, queue, region, or human review process. The router may use simple rules, a classifier model, metadata from the application, or a combination of these.

Good routing starts with a small number of practical questions. Is this request urgent? Is it sensitive? Does it need a tool such as search or code execution? Does it require a specialist model? Is the preferred service healthy? What is the cost limit for this user or workflow?

Routing also improves resilience. If one model endpoint is overloaded, the router can shift eligible traffic to another endpoint. If a request is too risky for automation, the router can send it for review instead of forcing a model to guess.

A router reads each request and sends sensitive or risky work to review, and routine work to the right model tier.

Model tiering matches effort to difficulty

Model tiering is the practice of using different classes of models for different levels of work. A small model may be fast and cheap but less capable. A large model may handle subtle reasoning and complex language better, but it costs more and usually takes longer. A tiering policy decides when each is appropriate.

A common pattern is to start with the least expensive tier that can meet the quality target. If the request is simple, the small model answers. If the request is complex, sensitive, ambiguous, or fails a quality check, the system escalates to a stronger model. This is not about using weak models everywhere. It is about saving the strongest models for the work that needs them.

Tiering can be based on request type, user plan, document complexity, confidence score, evaluation result, or business importance. For example, a small model might classify incoming support tickets, a mid-tier model might draft a reply, and a top-tier model might handle legal or executive communication only after extra checks.

✅

The best tier is contextual. A small model with the right prompt, data, and checks can outperform a larger model used carelessly.

The three controls work best together

Caching, routing, and tiering are often discussed separately, but in a real system they form one control plane. The cache can avoid a route entirely. The router can choose which cache to check, which model tier to use, and whether a fallback is allowed. The tiering policy can decide when a low-cost answer must be verified by a higher tier.

Consider an employee asking a policy question. The gateway authenticates the user, commonly through an identity provider such as Microsoft Entra ID. The system checks whether a valid answer already exists for that user’s permission scope and document version. If not, the router may retrieve approved policy documents, send the request to a mid-tier model, and escalate only if the answer has low confidence or touches a regulated topic.

In a platform such as Microsoft Foundry, the same design principles still apply. The names of the endpoints and management tools may differ, but the architecture should still make reuse, routing, tier selection, identity, logging, and evaluation explicit rather than hidden inside application code.

The gateway authenticates the user, checks the cache, then routes to a model tier and escalates only when needed.

The real design choices leaders and practitioners face

The central leadership question is not whether caching, routing, and tiering are clever. It is where the organization wants to sit on the trade-off curve between cost, speed, quality, and risk. A customer support chatbot, a clinical assistant, a code review tool, and a financial reporting workflow should not share the same policy just because they all call models.

Practitioners need concrete service level objectives. A service level objective, or SLO, is a measurable target for behavior such as response time, success rate, or error rate. For example, a team may require that 95 percent of routine answers return in under two seconds, that regulated requests never use semantic caching, and that any low-confidence answer escalates before it reaches a customer.

Leaders also need ownership. Someone must decide which data may be cached, how long it is retained, who can see logs, what counts as acceptable quality, and when cost savings are not worth the risk. Without those decisions, engineers are forced to encode business policy by guesswork.

Decide which workflows optimize for speed, which optimize for accuracy, and which optimize for auditability.
Set separate policies for public content, internal content, customer data, and regulated data.
Budget by workload rather than by model, because routing and tiering can change which model serves a request.
Measure user-visible outcomes, not only infrastructure metrics.
Make fallback behavior explicit before the preferred model or region fails.

📌

Policy beats preference. A written routing and tiering policy prevents every team from inventing a different hidden standard.

Failure modes to design out early

A cache can return a stale or unauthorized answer. A router can misclassify a request and send it to the wrong model or region. A tiering policy can save money while quietly lowering quality. These are not reasons to avoid the techniques. They are reasons to design them with tests, limits, and observability from the beginning.

Start with safe defaults. If the cache key is uncertain, miss the cache and compute a fresh answer. If the router is unsure, choose the safer path rather than the cheaper one. If a lower tier produces an answer that fails validation, escalate or decline rather than pretending confidence is higher than it is.

Log the decision trail without exposing sensitive content unnecessarily. A useful trace records the cache decision, route chosen, model tier, prompt version, retrieval sources, latency, cost, quality checks, and fallback events. This gives engineers the evidence they need to tune the system and gives risk owners a basis for audit.

Cache poisoning: a bad or malicious answer is stored and then reused.
Permission leakage: an answer created for one user or group is returned to another.
Silent degradation: cheaper tiers reduce quality without visible alerts.
Retry storms: failed requests are retried so aggressively that they overload the system.
Unbounded fallback: traffic shifts to a more expensive model without budget controls.

⚠️

Never cache authorization away. A cached answer must still be checked against the current user, tenant, permissions, and source data rules.

Key takeaways

Caching saves time and cost only when the system can prove an old answer is still safe to reuse.
Routing is the policy point that sends each request to the right model, tool, region, queue, or review path.
Model tiering reserves larger and more expensive models for work that genuinely needs their capability.
The strongest designs combine cache rules, routing rules, tier rules, identity, evaluation, and logging in one control plane.
Leaders must set explicit trade-offs for speed, cost, quality, privacy, and auditability before engineers can implement reliable policy.
Safe defaults matter: when uncertain, miss the cache, choose the safer route, or escalate to a stronger check.

Glossary

Cache: A store of previous results that can be reused when a new request is validly treated as the same or similar work.
Cache key: The set of values used to decide whether a cached result matches a new request.
Semantic cache: A cache that reuses results based on similarity of meaning rather than exact text match.
Router: A decision point that sends a request to the most appropriate model, tool, service, or review path.
Model tier: A class of model chosen for a particular balance of capability, speed, cost, and risk.
Inference request: A request that asks a model to produce an output from given input.
Service level objective: A measurable target for how a system should behave, such as response time, availability, or error rate.
Fallback: A planned alternate path used when the preferred path is unavailable, too slow, too risky, or fails a quality check.