Marin 535B-A23B Architecture Postmortem: MoE, XSA, MuonH and 18T-Token Curves
StasBekman · x · 2026-11-13
Marin published detailed training notes on the architecture decisions behind 535B-A23B, pretrained on 18T tokens — packed with plots and decision notes for practitioners. Highlights: Latent MoE (8 of 384 experts, 2 shared, SwiGLU); 3:1 sliding-window/global attention with GQA and gated attention; partial RoPE on local and NoPE on global layers; MuonH/AdamH Hyperball optimizers; exclusive self-attention (XSA); kernel-4 short convolutions; top-k quantile balancing routing; a wide-shallow 6144-hidden/48-layer design (128 width:depth); no leading dense layers. Full per-step loss curves mark the ragged all-to-all EP switch, data-mix updates, and 4k→16k→65k context extensions.
More from Infra
- llama.cpp gotcha: -ngl -1 means auto, not all — switching fixed CPU usage and 60→76 t/s — randomjapaneselearn · 2026-11-13
- MoE expert offloading on a 16GB RX 9070XT: three engines benchmarked, ExLlamaV3 leads — willeyh · 2026-11-13
- Microsoft data center outage sparks jokes about servers 'going on protest' — SuB8u · 2026-11-13
- Data centers need power – lower income, rural residents pay the price — PaulHoule · 2026-11-13
- Notion indexes 10B+ docs on turbopuffer, cuts search costs 80% — DevinLewtan · 2026-11-13
- Even cutting all China GPU purchases leaves massive unmet compute demand — peterjliu · 2026-11-13