ByteDance releases sparse autoencoders for Qwen3-30B revealing interpretable MoE expert specializations
_akhaliq · x · 2026-11-05
ByteDance Seed released sparse autoencoders for Qwen3-30B-A3B on Hugging Face (ByteDance-Seed/Qwen3-30B-A3B-MoE-SAE), alongside a paper showing that MoE experts exhibit human-interpretable fine-grained specializations.
Technical details: TopK SAEs (k = 100, 2^20 latents) trained on the MoE block output at layers 4, 30, and 46; each SAE is 17.2 GB in float32, Apache-2.0 licensed. The model card includes full loading/encode-decode code that operates directly on raw MoE outputs before residual addition. A rare large-scale open resource for MoE interpretability research.
More from Models
- ThursdAI podcast roundup: AI edits AI, 948 Linux bugs, Kimi K3.1 weights — thursdai_pod · 2026-11-06
- Dev complains Claude Code quality has tanked: same models, shockingly worse results — IanArawjo · 2026-11-06
- Cryptographer Matthew Green: can abliterated GLM write 0days? Just test it — matthew_d_green · 2026-11-06
- Astra refusals decoded: safety training fires at the local level, not against safety research — thebasepoint · 2026-11-06
- Developer: GPT 6.1 is far less creative than Opus 5.5, taste must be spelled out — zsakib_ · 2026-11-06
- Chip design lab Ricursive releases open Ripple 65B, claims it beats US open models for $32M — Sam Witteveen · 2026-11-06