Live labels from the forward pass teach MoE models to report their own routing
jonasgeiping · x · 2026-10-21
Geiping explains that routing decisions in an MoE are not captured by offline training or RL, so models normally can't access them. But live labels can be picked directly from the forward pass, letting researchers train MoE models to verbalize these internal states — and they do so very well.
More from Research
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21
- AI drug discovery reality check: 'give me everything that binds,' then filter for toxicity — burny_tech · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21