Meta Muse Glimmer 30B Native 512k Context: Architecture and Benchmarks
mr_il · reddit · 2026-08-17
The author extended Muse Glimmer 30B's context window to 512k tokens by simply changing a config value, revealing its unique architecture and testing it across benchmarks.
Architecture Insights:
- Full GQA attention layers have no positional encoding; only 2048-wide SWA layers use RoPE.
- This design infers long-distance relationships, making it uniquely suited for context extension.
Benchmark Results (512k tokens):
- Needle in a Haystack (1/4 needles): 100%
- Multi-hop retrieval: 100%
- Semantic lookups: 100%
- Memory (Agentic session): 100% (up to 385k)
- LongBench v2: No performance degradation
- LongCodeQA: No performance degradation
- Counting instances: Performance degrades at extreme lengths (60% at 385k, 22% at 512k).
Code and reports are available on GitHub.
More from Models
- Qwen 3.8 scores high on WeirdML but uses many reasoning tokens — teortaxesTex · 2026-08-17
- OpenAI's Astra hit Critical capability threshold in cyber, parts of development paused — johnseach · 2026-08-17
- HF's Lewis Tunstall: we use closed models all the time, guardrails forced GLM switch — _lewtun · 2026-08-17
- Will hosted models remain useful with increasing watermarks and limits? — vboykis · 2026-08-17
- Meta releases Muse Code; Alibaba open-sources Qwen3.8-27B — emmanuelvivier · 2026-08-17
- Meta open-sources 30B-parameter agent model Muse Glimmer under Apache 2.0, runs on consumer GPUs — emmanuelvivier · 2026-08-17