Serving MoE — Mixture-of-Experts at Scale

A mixture of experts model, often shortened to MoE, is built on a simple bet: for any one word or fragment of text, only a small part of the model needs to think hard. That can make very large models practical, because each request activates only a few specialized submodels instead of every parameter. Serving MoE is the work of turning that promise into reliable responses for real users, where latency, cost, memory, and failures matter as much as raw model quality.

Start with the job of a language model

A language model receives text, breaks it into tokens, and predicts what token should come next. A token is a small unit of text, such as a word, part of a word, or punctuation mark. The model does this prediction many times in a row to produce an answer.

In a dense model, nearly all of the model’s learned weights are used for every token. Weights are the stored numbers that define what the model has learned. If the model is large, this can produce strong answers, but it also means every generated token must pay the compute cost of a large model.

A mixture of experts model changes one part of this pattern. It contains multiple expert networks, and a small routing network chooses which expert or experts should process each token. The model may have many total parameters, but only a fraction are active for any single token.

What serving means

Serving is inference in production. Inference means running a trained model to produce outputs, not changing the model through training. Production means real users or systems are waiting for answers under practical limits: time, money, availability, and safety.

Serving a model is not just calling a function. The system must accept requests, tokenize input, schedule work on hardware, run the model, stream or return outputs, record measurements, and handle overload or failure. For a chat system, this happens repeatedly as every new output token becomes the input for the next step.

MoE serving adds one important difference: the system does not know exactly which experts will be needed until the router sees the tokens. That makes capacity planning and scheduling harder than for a dense model with a fixed compute path.

A serving system accepts a request, tokenizes it, schedules work on GPUs, and streams tokens back.

The MoE hot path

Inside an MoE layer, the router scores the available experts for each token. In many systems it chooses the top k experts, where k is a small number such as one or two. This is called top-k routing. The selected experts process the token, and their results are combined before the model continues.

This sounds like a local choice, but in a serving system it affects the whole machine. Experts are often spread across multiple graphics processing units, or GPUs, which are processors designed for highly parallel numerical work. If a token is assigned to an expert on another GPU, the system must move data across the hardware fabric.

For one token, that transfer may look small. For thousands of tokens from many users, transfers become a central part of the serving cost. MoE serving is therefore a coordination problem as much as a compute problem.

The hidden tax: routing, imbalance, and communication

Sparse compute is the attraction of MoE: only some experts run for each token. But sparsity creates a new tax. Tokens must be grouped by expert, moved to the right place, processed, and then put back into the original order. This movement is often called dispatch and combine.

When experts live on different GPUs, systems commonly use all-to-all communication. All-to-all means many devices send data to many other devices at the same time. It can be efficient on the right hardware, but it is sensitive to network bandwidth, congestion, and small scheduling delays.

Load imbalance is another tax. If the router sends too many tokens to one expert and too few to another, the busy expert becomes the slowest step. Some systems use a capacity limit per expert. Tokens beyond that capacity may be delayed, rerouted, or dropped depending on the model design and serving policy.

⚠️

Sparse is not free. MoE reduces active compute, but routing and communication can erase the gain if the serving system is poorly matched to the model.

Sparse routing groups tokens by expert, dispatches them, processes, then recombines in order.

Memory decides what is possible

Serving large models is often limited by memory before it is limited by arithmetic. The model weights must fit somewhere, and an MoE model can have many more total weights than a dense model with similar active compute. Experts may be sharded across GPUs, meaning different devices hold different parts of the model.

There is also runtime memory. During generation, the system stores a key-value cache, often called a KV cache. The KV cache holds intermediate attention data so the model does not have to recompute the full prompt for every new token. Longer prompts, larger batches, and more simultaneous users all increase this cache.

MoE adds placement questions. If the right expert is not on the right device, the system pays communication cost. If too many experts are replicated everywhere, the system pays memory cost. Good serving design usually balances expert placement, batch size, cache management, and the expected traffic pattern.

Replicate small or frequently used components when memory allows.
Shard large expert weights when total model size exceeds one device.
Batch requests carefully so GPUs stay busy without delaying short requests too much.
Track KV cache growth because long conversations can crowd out new work.
Place experts with hardware topology in mind, not just by parameter count.

Sharding experts saves memory but adds hops; replicating them cuts hops but costs memory.

Operating for predictable latency

Users notice latency, not parameter counts. Latency is the time between a request and a useful response. In language model serving, two measurements matter especially: time to first token, which is how long the user waits before output begins, and tokens per second, which is how quickly the answer streams after that.

MoE systems can show uneven latency because different requests activate different experts. A batch of requests may look cheap on average while one expert is overloaded. Production systems therefore need admission control, queueing policies, expert-level metrics, and overload behavior that is chosen deliberately rather than discovered during an incident.

A service-level objective, or SLO, is an internal target for reliability or latency. A service-level agreement, or SLA, is a commitment made to a customer or partner. MoE serving should be designed around these targets, because the cheapest configuration at low traffic may be the least predictable one at peak traffic.

The decisions leaders and practitioners face

The first decision is whether MoE is the right model shape for the workload. MoE is most attractive when model quality benefits from many parameters but each token can use a small subset. It is less attractive when traffic is small, hardware is limited, or predictable low latency matters more than peak efficiency.

The second decision is how much system complexity the team can operate. Dense models are simpler to serve because every token follows a more uniform path. MoE can improve cost or quality, but it asks for stronger scheduling, monitoring, capacity planning, and failure handling.

The third decision is where to spend the optimization budget. A team may get more benefit from better batching, cache management, or expert placement than from changing hardware. Another team may be limited by network bandwidth and need a different cluster layout. The right answer depends on measurements from the real workload.

📌

Ask for traces. Before committing to an MoE serving design, inspect token-level traces that show expert choices, communication time, queueing time, and cache use.

Key takeaways

MoE models activate only a few experts per token, which can reduce compute without reducing total model size.
Serving MoE is harder than serving a dense model because routing choices create variable work and communication.
Memory pressure comes from both expert weights and the KV cache used during generation.
All-to-all communication and expert load imbalance are common sources of lost performance.
Reliable MoE serving requires operational controls such as admission limits, expert-level metrics, and overload policies.
The best MoE serving design depends on real traffic, latency targets, hardware topology, and the team’s ability to operate complexity.

When predictable low latency dominates, a dense model is simpler; otherwise MoE can add capacity.

Glossary

Mixture of experts: A model architecture that contains multiple expert networks and uses a router to activate only a small subset for each token.
Expert: A specialized subnetwork inside an MoE model that processes tokens selected for it by the router.
Router: The small network or function that scores experts and chooses which ones should handle each token.
Token: A small unit of text, such as a word, part of a word, or punctuation mark, that a language model processes.
Inference: The act of running a trained model to produce outputs rather than training it to change its weights.
Top-k routing: A routing method that sends each token to the k highest-scoring experts.
All-to-all communication: A communication pattern in which many devices send data to many other devices at the same time.
KV cache: Stored key-value attention data that lets a language model generate new tokens without recomputing the whole prompt.
Service-level objective: An internal target for a service’s reliability, latency, or other measurable behavior.