Marin 535B-A23B finishes 18T-token pretraining with zero instabilities, architecture detailed
ChengleiSi · x · 2026-11-10
Open Athena's Marin project publishes a full architecture retrospective for Marin 535B-A23B, which finished pretraining on 18T tokens on Nov 5 — live-streamed, publicly forecast, with zero instabilities.
Scale: just 6 full-time employees; B200 stack stood up in a month, GPU scale-up of 300x. The author credits the published ecosystem (Llama, DeepSeek, Qwen, Gemma, OLMo, Cohere, Nemotron, Arcee) for making a first large-scale run this smooth.
Architecture highlights:
- Shallow wide stack: 48 layers, hidden 6144 (width:depth 128), no leading dense layers; every block is Latent MoE (8/384 experts, 2x latent compression, 2 shared experts, sigmoid router, top-k weights renormalized to 2.5)
- 3:1 local sliding-window (2048) / global attention with GQA, gated attention, partial RoPE (local) / NoPE (global)
- Notable components: learnable latent RMSNorm, GatedNorm, exclusive self-attention (XSA), fixed-1.3 QK RMSNorm, short convolutions (kernel 4), top-k quantile balancing on raw logits
- Muon/Hyperball optimizer, 1% warmup then linear decay to 5%; loss curve annotated with EP switch, two data-mix phases, 4k→16k→65k context extensions
More from Infra
- DSpine explained: 5-layer draft model guesses 16 tokens in one forward pass — karminski3 · 2026-11-10
- Fireworks CEO discloses October breach: third-party cloud access led to customer data impact — lqiao · 2026-11-10
- Report: Anthropic pushing governments to require firmware locks blocking unapproved models on Western GPUs — rickasaurus · 2026-11-10
- Cerebras releases CS-4 engineering documentary detailing wafer-scale AI system design — draecomino · 2026-11-10
- Semi-Apocalypse podcast: market panic, enterprise AI, and the 2028 GPU glut theory — BenBajarin · 2026-11-10
- Ben Bajarin: supply chain bottleneck analysis is back at center stage — BenBajarin · 2026-11-10