Home

Mixture-of-Experts at Scale

Why the biggest models are mostly switched off. Sparse mixture-of-experts from the ground up — active vs total parameters, top-k routing and load balancing, expert parallelism, and why MoE is cheap to train but awkward to serve.