Expert Parallelism — Mixture-of-Experts at Scale

A neural network is usually imagined as one large calculation: data goes in, every layer does its work, and an answer comes out. Expert parallelism begins with a different question: what if only a small part of the model needs to work on any one piece of data? That idea lets a model contain many more parameters, which are its learned numbers, without paying the full computing cost on every request. The price is coordination. The system must decide which specialist parts, called experts, should handle each item, move data to wherever those experts live, and bring the results back fast enough that the savings are real.

Start with one calculation

A neural network transforms input into output through layers of arithmetic. In a language model, the input is first split into tokens, which are small units such as words, word pieces, or punctuation. Each layer updates a numerical representation of each token so the model can predict or generate the next token.

In a dense model, almost all parameters in a layer are used for every token. If the layer has a billion learned numbers, each token effectively asks that large layer to participate. Dense models are simple to reason about and easier to run predictably, but as they grow, every step becomes more expensive.

A sparse model changes the bargain. It may contain many experts, but each token uses only a few of them. The model can have a large total capacity while activating only a smaller fraction of that capacity at a time.

📌

Sparse is not small. A sparse model can be very large in total size while using only a small part of itself for each token.

What an expert is

An expert is a subnetwork inside a larger model. It is not usually a human-readable specialist like “legal expert” or “math expert,” though people sometimes use that language informally. It is a learned function that becomes useful for some patterns of tokens because training nudges it that way.

Most expert systems use a router. The router is a small learned component that scores the available experts for each token. The model then sends the token to the top one, top two, or another small number of experts. The chosen experts process the token, and their outputs are combined before the token continues through the rest of the model.

This design is commonly called a mixture of experts, or MoE. The word mixture matters: the final output is not owned by one expert alone. It is a combination of routed expert work and the shared layers around it.

A router scores experts for each token, sends it to a chosen few, then combines their outputs.

What expert parallelism adds

Parallelism means doing work at the same time. In machine learning systems, the work is often split across graphics processing units, or GPUs, and sometimes across other accelerators. Expert parallelism is the specific practice of placing different experts on different devices so tokens can be sent only to the devices that hold the experts they need.

This differs from data parallelism, where each device has a full copy of the model and processes different examples. It also differs from tensor parallelism, where one large mathematical operation is split across devices. Expert parallelism splits the model by expert: device A may hold experts 1 through 8, device B experts 9 through 16, and so on.

The benefit is memory and capacity. A single device does not need to hold every expert, so the model can contain more experts than would fit on one device. The cost is movement. Tokens must travel across the network to reach their selected experts, and their results must travel back.

Data parallelism copies the model, tensor parallelism splits one op, expert parallelism splits by expert.

The hidden cost is communication

In a dense layer, tokens usually stay on the device that already has the layer’s weights, or they move in a predictable pattern. In an expert-parallel layer, routing can send tokens from many devices to many other devices. This many-to-many exchange is often called all-to-all communication.

All-to-all communication can dominate the runtime if the network between devices is slow, congested, or poorly matched to the routing pattern. The arithmetic inside each expert may be efficient, yet the whole layer can stall while devices wait for tokens to arrive.

This is why expert parallelism is a systems design, not just a model design. The best number of experts, the number selected per token, the batch size, and the physical placement of devices all interact. A model that looks efficient on paper may disappoint if the communication fabric cannot keep up.

More experts can increase model capacity, but they also create more possible destinations for tokens.
Selecting more experts per token can improve quality, but it increases computation and network traffic.
Larger batches can make communication more efficient, but they may add latency for interactive serving.
Keeping frequently used experts near the tokens that need them can reduce movement, but routing patterns can change during training.

⚠️

Watch the network. Expert parallelism often fails first at the communication layer, not at the arithmetic layer.

On a slow fabric all-to-all traffic dominates; on a fast fabric expert compute is the limit.

Training teaches both experts and router

During training, the model sees examples and adjusts its parameters to reduce error. In a mixture of experts model, training must improve the experts, the shared layers, and the router at the same time. The router is especially important because it controls which experts receive learning signals from which tokens.

A common problem is imbalance. If the router sends too many tokens to one expert, that expert becomes overloaded while others sit idle. The overloaded expert may also learn too broadly, while underused experts learn too little. To prevent this, training often includes load-balancing losses or routing rules that encourage tokens to spread across experts.

There is also a capacity limit. Each expert can accept only a certain number of tokens in a step. If too many tokens choose the same expert, the system must either drop some token-expert assignments, reroute them, or wait. Each option has consequences for quality, speed, and stability.

Inference changes the priorities

Inference is the use of a trained model to produce outputs. In training, the system is judged by throughput and learning stability. In inference, especially for chat or search-like applications, latency becomes central: users notice how long the first token takes and how quickly later tokens arrive.

Expert parallelism can help inference because each token activates only a few experts. But the router’s decision can send a token across machines, and that trip may take longer than the expert computation itself. The serving system must manage this carefully, often by batching requests, caching shared state, and placing experts to reduce costly hops.

There is a practical tension between utilization and responsiveness. Waiting briefly to form a larger batch can make GPUs busier and reduce cost per token. Waiting too long makes users see delay. Expert-parallel serving is therefore an exercise in choosing the least harmful queueing, not eliminating queueing entirely.

✅

Good fit. Expert parallelism is strongest when the saved computation is larger than the added routing and communication cost.

Decisions leaders and practitioners face

The first decision is whether the workload actually needs expert parallelism. If the target model fits comfortably on available devices and dense computation meets cost and latency goals, expert parallelism may add complexity without enough benefit. Its value rises when total model capacity is the constraint, not merely raw compute.

The second decision is where to spend the complexity budget. Teams must choose the number of experts, how many experts each token uses, how experts are placed across devices, and which other forms of parallelism to combine with expert parallelism. These choices affect training stability, serving latency, failure recovery, observability, and operational skill requirements.

The third decision is organizational. Expert-parallel systems require model engineers, distributed systems engineers, and infrastructure operators to work from the same measurements. A routing change can alter network traffic. A network topology change can alter model throughput. A batching policy can alter perceived product quality. Treating these as separate concerns leads to slow debugging and unreliable planning.

Use expert parallelism when model capacity or device memory is the binding constraint.
Budget for network bandwidth, routing metrics, and failure testing from the start.
Measure both tokens per second and user-facing latency; either one alone can mislead.
Plan for combined parallelism, because large systems often use expert, data, tensor, or pipeline parallelism together.

Expert parallelism pays off when memory capacity is the constraint, not raw compute.

Failure modes to recognize early

The clearest failure mode is expert collapse, where the router favors a small set of experts and leaves others underused. Collapse wastes parameters and can make training unstable. It is usually visible in routing statistics before it is obvious in model quality.

Another failure mode is communication saturation. Devices appear underused because they are waiting, not because there is too little work. This can look like a model problem when it is really a placement, batching, or network problem.

A subtler failure mode is operational fragility. If one device holding popular experts slows down or fails, the impact can be larger than its share of the fleet suggests. Production systems need backpressure, retries, capacity planning, and clear fallbacks, because sparse routing creates uneven pressure by design.

Key takeaways

Expert parallelism places different experts on different devices and routes each token only to the experts it needs.
It is most useful for sparse mixture of experts models, where total parameter count can grow faster than per-token computation.
The main tradeoff is saved computation versus added all-to-all communication.
Training must balance expert use so the router does not overload a few experts and starve the rest.
Inference depends on careful batching, placement, and latency management, not only on model architecture.
Adopting expert parallelism is a joint model, systems, and operations decision.

Glossary

Expert parallelism: A way to run a sparse model by placing different experts on different devices and sending tokens to the devices that hold their selected experts.
Expert: A learned subnetwork inside a larger model that processes some tokens chosen by a router.
Mixture of experts: A model design in which a router selects a small number of expert subnetworks for each token and combines their outputs.
Router: The learned component that scores experts and decides which ones should process each token.
Token: A small unit of input or output, such as a word, word piece, character, or punctuation mark.
Dense model: A model that uses most or all of its relevant parameters for each input rather than selecting a small subset.
All-to-all communication: A many-to-many data exchange in which devices send data to many other devices and receive data from many other devices.
Inference: The use of a trained model to produce outputs for new inputs.