Leaked Architecture of ~400B MoE Model with Aggressive GQA Sparks Interest
teortaxesTex · x · 2026-08-10
A developer analyzed the architecture of a suspected new model, Muse Glimmer 30B. It is presumed to be a Mixture of Experts (MoE) model with around 400B total parameters and 15B active parameters. The architecture notably features a 3:1 local/global attention ratio paired with absurdly aggressive Grouped-Query Attention (GQA). Observers are amazed that such an extreme design works remarkably well.
Related event: Leaked Architectures of New 30B to 400B Parameter Models Spark Discussion(2 posts)→
More from Models
- Opus 5.5 turns heads with surprisingly polished vertical video demo — FinanceYF5 · 2026-10-16
- Astra 6.1 may arrive before end of month, reads dev's 'we are cooking' tease — haider1 · 2026-10-16
- Open-source OCR model indra-ocr trends on Hugging Face with CPU support — datalab-to · 2026-10-16
- Xiaohongshu's Vela 35B turns anonymized user behavior into supervision, gains +11.5 on VelaBench — AdinaYakup · 2026-10-16
- KOL: No better model than Anthropic's Opus right now, its magic is back — kimmonismus · 2026-10-16
- Users blast Gemini in Chrome's 'draconian' usage caps: memory costs 2-4% per use — North-American · 2026-10-16