DOT-MoE: converting dense models into sparse MoE via optimal transport
burkov · x · 2026-10-16
LLMs are expensive to run since every parameter fires for every token. Sparse MoE avoids this, but training from scratch is costly and unstable. Converting dense models into MoEs (MoEfication) is appealing, yet current methods rely on heuristics — random splits, weight similarity, or activation clustering — ignoring how neurons affect outputs. DOT-MoE reformulates neuron assignment as an optimal transport problem, shipping neurons to experts under strict capacity constraints (each expert gets exactly the same... [truncated]).
More from Infra
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16
- Samsung cuts Q4 phone output as Nittobo shifts E-glass to AI-chip T-glass — zephyr_z9 · 2026-10-16
- White House AI chief discusses Empire AI, New York's shared compute infrastructure — mkratsios47 · 2026-10-16