Meta Muse Glimmer 30B Native 512k Context: Architecture and Benchmarks
mr_il · reddit · 2026-08-17
The author extended Muse Glimmer 30B's context window to 512k tokens by simply changing a config value, revealing its unique architecture and testing it across benchmarks.
Architecture Insights:
- Full GQA attention layers have no positional encoding; only 2048-wide SWA layers use RoPE.
- This design infers long-distance relationships, making it uniquely suited for context extension.
Benchmark Results (512k tokens):
- Needle in a Haystack (1/4 needles): 100%
- Multi-hop retrieval: 100%
- Semantic lookups: 100%
- Memory (Agentic session): 100% (up to 385k)
- LongBench v2: No performance degradation
- LongCodeQA: No performance degradation
- Counting instances: Performance degrades at extreme lengths (60% at 385k, 22% at 512k).
Code and reports are available on GitHub.
More from Models
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- LLM-as-a-Verifier lifts Nemotron 3.5 Super by 9-10 points on agentic benchmarks — simonguozirui · 2026-10-16
- Bonsai 2 ternary models tested: 4B is the sweet spot, but 27B unexpectedly fumbles file writes — cephaloform · 2026-10-16
- Meta's Omnilingual ASR, covering 1,600 languages, named a TIME Best Invention of 2026 — AIatMeta · 2026-10-16
- Apollo Research: GLM-5.3 shows unprecedented 'update-aware reasoning' — repligate · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16