xavier-ramirez.com
STAGE 0 · 2·B · 3 NODES

Mixture of Experts

Frontier models are mostly Mixture-of-Experts. Instead of running one giant feed-forward network for every token, a router sends each token to a few specialist experts — so the model can hold far more knowledge while each token costs about the same to run.

THE 3 STEPS
01Router + top-K02Load balancing03The shared expert
NODE 01 / 03

One router, many experts

A Mixture-of-Experts layer is a pile of small feed-forward networks — the experts — plus a router that picks which few to run for each token.

  • Last lesson, one big feed-forward network ran for every token. MoE replaces it with many smaller ones, the experts.
  • *A tiny router scores the token against each expert and sends it to only the top few — top-K*. The rest stay switched off for that token.
  • More experts means more total knowledge at the same cost per token. An 8-expert, top-2 layer holds about 4× the feed-forward parameters of a dense one, while each token still runs just 2 experts — roughly double the work.
  • The green bars are the router's scores. Change the token and the top experts change; raise top-K to run more of them.
TOKEN → ROUTER → TOP-K EXPERTS

Switch the token, then raise or lower top-K — watch which experts light up.

A code token leans on the experts that specialised on code — a different set than a plain word picks.

ROUTING
EXPERTS8
ACTIVE2
PARAMS USED2/8 · 25%
How many experts exist, how many run for this token, and the share of the layer's feed-forward parameters that touches — the rest stay idle.
NODE 02 / 03

Stop everyone piling onto one expert

Left to itself, the router plays favourites — a few experts get swamped while others sit idle and never learn.

  • Nothing forces balance. Early in training the router finds a few experts it likes and keeps sending tokens there.
  • Swamped experts overflow. Each expert has a capacity — a cap per batch. Tokens over the cap are dropped: they skip the layer entirely (shown in red).
  • Idle experts are wasted parameters — and they never improve, because they almost never see a token.
  • *An extra load-balance loss fixes it.* It penalises lopsided routing, nudging the router to spread tokens evenly. Slide it up.
GOOD TO KNOW · CAPACITY
Why tokens get dropped
Each expert gets a fixed number of slots per batch — usually 1.25× the average, the capacity factor. It keeps the maths a fixed shape the GPU can run efficiently. If routing is lopsided, popular experts run out of slots and the extra tokens are skipped for that layer — a real cost of poor balancing.
TOKENS PER EXPERT · ONE BATCH

Drag the balance strength up from 0 — the load evens out and the dropped (red) tokens vanish.

No balancing: a couple of experts hoard the tokens, overflow their capacity, and the rest sit idle.

LOAD
BUSIEST174 / 80
IDLE2 / 8
DROPPED137 / 512
The most-loaded expert, how many experts are nearly idle, and how many of the batch's 512 tokens overflowed capacity and were dropped.
NODE 03 / 03

One expert that's always on

Some knowledge — grammar, punctuation, the most common words — every token needs. Routing that to a specialist wastes the router's picks.

  • Without a shared expert, every routed expert relearns the common stuff — the same basics duplicated across all of them (the grey band).
  • *A shared expert runs for every token, always* — no routing decision. It holds the common knowledge once.
  • Now the routed experts are free to specialise on the rarer, harder patterns, and a top-K slot isn't burned on the basics.
  • This is the modern design (DeepSeek-V3 and others): one always-on shared expert alongside the routed ones.
SHARED EXPERT + ROUTED EXPERTS

Toggle the shared expert off — each routed expert sprouts a grey 'common' band, the duplicated basics.

Shared expert on: the common knowledge lives in one always-on expert (green), so the routed experts hold only their specialty.

EXPLAIN IT BACK
A Mixture-of-Experts model can hold many times the parameters of a dense model but cost about the same to run per token. How?
NEXT: CONSTRAINED DECODING

Every layer so far shapes the numbers that become the next-token scores. Next, constrained decoding: forcing those scores to obey a grammar so the model can only emit valid JSON, SQL, or whatever schema you demand.

← Back: FlashAttention & GQAContinue to constrained decoding →
Language: English