The premise is Option A: the router is never told which task a token belongs to. It sees only content and must discover any task-aligned routing itself. The real signal throughout is mutual information between the routing path and the task label, always read against a shuffled-label control — structure only counts if it beats the null.
The work ran in four phases: a synthetic task where the ground truth is known exactly; an extension to real text; — when the text results turned out ambiguous — a nine-experiment study of why the router behaves the way it does; and a follow-up that replaces per-layer experts with one shared pool and an adaptive-depth head. This summarizes all four and states plainly what was and was not established.
Part I
Synthetic: modular arithmetic
Arithmetic mod P = 97, with the operation playing the role of a hidden task cluster on a K ∈ {2, 4, 8} diversity ladder. Trained in a grokking regime (weight decay 1.0) so the network is forced to generalize rather than memorize. Backbone: a residual MLP tower, d=128, four layers, E=2K experts per layer, one active, per-layer Gumbel-softmax router.
What happened
- Self-organization is real. With no operation signal to the router, MI(path; op) reaches 0.93–1.0 bits at K=2 and 2.0 bits — the ceiling — at K=4, against roughly zero for shuffled labels.
- The mechanism is legible. The four layers run a fixed three-stage algorithm: L0 splits per operation; L1 merges into families the design never specified ({add,sub} share an expert, {mul,div} share another); L3 is an answer read-out that routes by the computed result, not the operation (MI with the result ≈ 1.3 bits vs ≈ 0.1 with the operands).
- At K=8, sparse beats dense at matched active compute. This is the first task size where the margin opens — K ≤ 4 were ties because one dense expert already solves them.
| arm | val acc | note |
|---|---|---|
| dense (E=1) | 0.989 | baseline |
| sparse, vanilla | 0.989 | tied — no margin without the gate |
| sparse + alive-gate | 0.994 | every seed beats every dense seed |
| oracle (tight) | 0.998 | routing handed the ground truth |
| random routing | 0.125 | chance |
The alive-gate — a per-expert learned scale with an L1 penalty, ramped in after grokking — is what surfaces the margin: it removes near-tie routing noise so the network can report its own per-layer width (e.g. [4, 2, 1, 2] of 32 slots at K=4).
Caveats. Three seeds, not the five-plus that reproducibility needs. The family groupings are seed-dependent: seed 0 finds the clean four-way split (MI = 3.0), seed 1 merges linear ∪ sympow onto one branch (MI drops to 2.75), seed 2 mixes cross-family pairs.
Tracing the metrics shows why: MI is flat from the moment grokking completes (~step 5000) through step 35000. Seed 1's information-losing merge was decided ~14000 steps before the sparsity penalty even begins to act. Which efficient decomposition a seed finds is locked during grokking itself — a metastable-minimum effect, not an under-optimized objective. More or stronger regularization cannot fix it.
Part II
Real text: the language-model extension
A nanoGPT-shaped byte-level LM (d_model=128, 4 layers, seq_len=128), experts in the FFN only, same routing machinery. Task: a hybrid corpus — four languages including Chinese and Japanese, literary prose and poetry, and Python / JavaScript source. Metric: bits per byte.
The phenomenon replicates
Sparse beats dense on every language, MI(path; source) climbs to near its ceiling against a flat shuffled control (2.67 / 3.0 vs 0.48 on the hybrid corpus; a linear probe reads source at 96.5%). The design doc's originally-planned contrastive path loss also works — MI to ceiling, no over-collapse, ~0.5% bpb cost. This is the GShard / Switch Transformer language-specialization result at a scale where every routing decision is visible.
But the controls deflate it
- A no-model baseline recovers most of it. A bag-of-byte-bigrams logistic probe — no network at all — predicts source at 0.83; the trained path vector manages 0.94 (chance 0.125). Much of the “specialization” is surface byte statistics.
- Substitution test: ~5 of 8 sources route on structure, not content. Swap a window's vocabulary but keep its punctuation and layout and routing does not move — for Python, JavaScript, prose and French. English routes almost entirely on punctuation: strip it and the probe falls from 0.97 to 0.05. Only German (and partly Spanish) route on lexical content that survives both controls — and even that is word identity, not meaning.
- At the token level, there are dedicated punctuation and whitespace experts. By layer 2 one expert is 65% punctuation (13× its corpus rate); a period routes to the same expert 100% of the time, in English, Japanese and Python alike. Layer 0 shows no such structure — it develops through depth.
This lands squarely with the skeptical wing of the current MoE-interpretability literature (the “standing committee” and “routing reflects geometry” results, and the polysemy / substitution method those tests are borrowed from). What looks like domain expertise is substantially surface form.
Part III
Why the router switches every token
Byte-level, per-token routing switches expert on ~80% of transitions — mean run length ~1.3 — and a single 128-byte window touches 74% of all expert-slots. At sequence granularity there is no parameter sparsity; the MoE behaves like structured dropout on a wide FFN. Seven attempts to change that, plus two follow-ons:
| intervention | run length | bpb | effect |
|---|---|---|---|
| 1 persistence penalty λ=0.03 | ~2.0 | 1.95 | near-free, but only helps CJK / code |
| λ=0.1 | ~25 | 2.03 | collapse — 5 experts, MI halved |
| 2 per-sequence coverage | — | — | window touches 74% of slots; no sparsity to find |
| 3 expert-count sweep | — | 2.15→1.94 | width helps ~10%; E=8 already = 85% of it |
| 4 byte-class span routing | — | — | fixed surface boundaries — relocates, doesn't fix |
| 5 router hysteresis | ~1.7 | 2.3 @ γ=8 | trained router's logit gaps (~10) defeat the bonus |
| 6 subword BPE tokens | 2.2 / 2.4 | 2.01 | experts become content-aligned; still a lookup |
| 7 router memory (prev + cross-layer) | 2.9 (L3) | 2.02 | CJK → one expert per window; prose flat |
| + drop the load-balance loss | 2.5 | 2.09 | prose content entropy 1.87→1.39; switches 61%→49% |
| + top-2 gating + shared expert | 2.0 (L3) | 1.80 | bpb ~10% better; routing more diffuse (3.4→4.6 experts per sequence) |
Verdict. Given the two things the router was missing — meaningful units (subword tokens) and state (previous-position and cross-layer routing) — it demonstrably can commit: Chinese and Japanese windows route to a single expert end to end. It chooses not to for English prose, which keeps switching every ~1.5 tokens.
So the fast switching is optimal, not a capability limit. A sentence interleaves function words, content words and punctuation, and those need different computation. The languages that commit do so because their token streams are long runs of one class. The load-balance loss makes it worse by forcing the router to fragment coherent groups to fill its expert budget. A DeepSeek-style shared expert — the published fix for specialists with no common ground — improves bits-per-byte ~10% but makes the routing more diffuse: it takes a third of the work and the routed experts switch even more freely. MoE at this scale is a wide FFN with an input-dependent activation mask keyed on the current position's surface class — correct behavior for heterogeneous text, just not the “modular specialists” picture.
Part IV
Shared expert pool and adaptive depth
A follow-up that changes the routing layer itself. Instead of every layer owning private experts, one pool of 28 experts is reused at every recursion step, with a single shared router that knows which step it is on. A per-token depth head then chooses how many steps — 2, 4 or 6 — each token gets, paying a ponder cost for depth. Both ideas are published (UniPool, Mixture-of-Recursions); what is new here is testing them together under Option A, with a shuffled-label control and a hand-routed oracle, on a task whose structure is known.
On K=8 arithmetic, 35,000 steps, three seeds per arm:
| arm | val acc | active MACs | depth |
|---|---|---|---|
| dense E=1, 4 layers | 0.989 | 274,560 | — |
| per-layer experts + alive-gate | 0.994 | 282,752 | — |
| 28 experts, 4 non-pooled layers | 0.995 | 278,144 | 4 |
| shared pool, fixed depth 4 | 0.994 | 288,896 | 4 |
| shared pool + adaptive depth | 0.998 | 150,656 | 2.0 |
The adaptive model solves K=8 arithmetic in two recursion steps and beats every other arm on every seed (0.998 against 0.994–0.995), at about 53% of the active compute — a win on both axes, not a tradeoff. The depth head trims every operation to the minimum it was offered, because two steps genuinely suffice: per-operation accuracy at depth 2 is 0.99–1.00. Parameter count is unchanged (~1.89M).
Why would less depth win? Four things compound. Weight-tied recursion is harder to optimize the deeper it runs, since gradients pass through the same weights N times. Training the readout at every depth first is deep supervision, which regularizes. Steps taken after the answer is formed add residual updates that don't help. And weight decay at 1.0 favors the smallest circuit that generalizes.
Expert reuse: a real loop, for one family
If the pool is shared, does the same expert fire at different steps? Reuse rate is the fraction of “could have been a new expert” slots that were repeats, per operation family, against a shuffled-family null.
| family | reuse rate | reproducible? |
|---|---|---|
| mul / div | 0.39 – 0.65 | yes, all six runs (null 0.07 – 0.26) |
| linear (add, sub, lin23) | 0.0 – 0.67 | no — seed-dependent |
| sympow (sq_sum, cube_sum) | 0.0 – 0.33 | no |
| max | 0.17 – 0.36 | no |
Multiplication and division examples route to the same expert at both steps 39–65% of the time — a learned loop, in the one family whose algorithm (a · b±1 mod P) applies a single transform twice. Every other family's occasional above-null reuse does not survive across seeds.
Does depth follow difficulty? No
All operations trim to depth 2 on all seeds. K=8 arithmetic has almost no per-operation difficulty spread — a single dense expert solves seven of eight operations at ~0.99 — so there is nothing for the head to differentiate.
The same holds on language. Ported to the transformer (one shared attention block plus a 16-expert pool, depth chosen per token), it reaches 1.95 bits per byte with 3.7M parameters, against 2.01 with 6.0M for the fixed four-layer model. But the head trims uniformly to ~4 steps for every token class — letters, whitespace, punctuation, digits and kanji all land within about 0.1 of 4.0 — and expert reuse is not source-specific (0.51–0.67 per source against a 0.58 shuffled null). Content heterogeneity, which per-token expert routing picks up, does not become compute-depth heterogeneity.
Caveats. Three seeds on arithmetic, one on the language model. The depth head only worked after the readout was trained at every depth early on: forcing full depth during grokking left the shallow readouts untrained, so the head had nothing valid to trim to and stayed at 6. Depth options were {2, 4, 6}, so 2 is the smallest depth tested; the true minimum may be 1.
Scorecard
Established
- Dense nets self-organize task-aligned routing with no task label, above a shuffled-label null.
- The structure is mechanistically legible at toy scale — the actual methodological contribution.
- At K=8 arithmetic, sparse + alive-gate beats dense at matched active compute, non-overlapping across seeds.
- On real text, the “specialization” is substantially surface form, and per-token switching is optimal for heterogeneous input.
- A shared expert pool with an adaptive-depth head solves K=8 arithmetic in 2 recursion steps at higher accuracy and ~53% of the compute (3 seeds), and the multiplicative family reuses one expert at both steps.
Not established
- No new mechanism. The alive-gate is L0-style gating; the contrastive path loss is SupCon on the routing decision; router memory is a light recurrence; shared pools and adaptive depth are published ideas (UniPool, Mixture-of-Recursions).
- Nothing that would change how MoE is built in practice.
- Single-seed throughout the LM work — not reproducibility-grade.
- Toy scale only; no scale-dependent claim, nothing shown to transfer.
- The one candidate-original idea was never checked against the literature.
Two threads with any freshness
Load-balancing manufactures routing scatter. Removing the load-balance loss concentrated English content-word routing from ~4 experts onto 2, and cut consecutive-content-word switching from 61% to 49%. It reframes load-balancing as actively fighting coherent specialization to satisfy a serving-capacity objective — concrete and mechanistic, worth a multi-seed check.
Dense-sufficiency predicts prunability. The alive-gate prunes freely on arithmetic (where one dense expert already suffices) and finds nothing on the LM task (where dense is measurably insufficient), across a 50× sweep of the penalty. The hypothesis: whether a single dense model already solves a task predicts whether its multi-expert structure is redundant. Plausible, untested against prior work — a literature search is the cheap next step and could settle whether this is a contribution or a re-derivation.
Assessment
This is a well-executed characterization study. It confirms known MoE behavior with unusually good mechanistic transparency — controlled ground truth, shuffled nulls, and full ablation access that billion-parameter interpretability work structurally cannot have — and it maps carefully why per-token routing behaves as it does at small scale.
That is a legitimate thing to have built. The nine-experiment switching study in particular is a clean, honest, multi-angle negative result, and the field's hype could use more of those. But it is not a paper with a novel claim, and none of it is a breakthrough.
The fork from here: wrap it up as understanding — capstone the switching study, run the literature check, call it done — or push one thread toward a contribution: multi-seed the load-balance finding, turn dense-sufficiency into a prospective predictive rule, and see whether either survives contact with the literature.
A standalone summary of a private research repository. Arithmetic results are 3-seed; all language-model figures are single-seed unless noted.