ByteDance Seed: Sparse AutoEncoders Reveal Human-Interpretable Fine-Grained MoE Expert Specialization
ByteDance-Seed · hf · 2026-11-06
ByteDance Seed researchers, with Shanghai University of Finance and Economics, published a paper using sparse autoencoders (SAEs) to probe what individual MoE experts actually do.
- Prior analyses count expert selection frequency on domain-labeled corpora, a coarse and fixed taxonomy that records selection rather than contribution.
- Since an MoE layer output is a router-weighted sum of expert outputs and the SAE encoder is affine, each latent feature's pre-activation can be decomposed into per-expert contributions.
- Findings: individual experts drive specific latent features, showing finer-grained and more concentrated specialization than routing statistics or domain labels suggest.
Paper: arXiv cs.LG, Nov 5, 2026.
More from Research
- Lightwheel and Synopsys bring CAE-grade contact physics into robot simulation — lukas_m_ziegler · 2026-11-06
- Lecture derives R-GCN's 4.75M-parameter bill and a 14.7× basis-decomposition fix — JeremyCMorgan · 2026-11-06
- Random steering vectors turn LLMs into teapots, integers and 2019 movies — Sauers_ · 2026-11-06
- Epoch AI review: OSWorld tasks are simple, unstable, and barely test GUI use — gleech · 2026-11-06
- One random vector in 240 makes Qwen3-30B believe it's a cat via MoE steering — Sauers_ · 2026-11-06
- Steering a single MoE expert at strength +1.25 rewires Qwen3-30B's persona — Sauers_ · 2026-11-06