Marin team explains why its 535B-A23B model looks the way it does, after a fully livestreamed 18T-token pretraining run
PMinervini · x · 2026-11-10
Percy Liang amplifies a design-rationale writeup for Marin 535B-A23B, which finished pretraining on 18T tokens last week. Author classiclarryd walks through the tradeoffs behind the model's size, 8-of-384 expert routing, GatedNorm, sconv, XSA, and a linear learning rate schedule.
Key points:
- The Marin project had just 6 full-time employees at Open Athena earlier this year, mixing AI veterans and newcomers
- They stood up a B200 GPU stack in a month, scaled compute 300x, and trained a 535B-parameter model
- The run was publicly forecast and livestreamed end-to-end, finishing slightly better than predicted with zero instabilities
Related event: Marin 535B completes 18T-token pretraining with full architecture write-up(2 posts)→
More from Companies & People
- Leaked: xAI Building Grok Academy at learn.grok.com with 15 Courses and Certificates — nima_owji · 2026-11-10
- Investor Issues Moratorium on 'Run Your Coding Agents in the Cloud' Startups — var_epsilon · 2026-11-10
- OpenAI shares lessons on closing the AI guidance gap in education across EMEA — OpenAI News · 2026-11-10
- Anthropic's $1.5T IPO: 2023 hires' $1M stock grants would be worth $78M, 78x gains — FinanceYF5 · 2026-11-10
- Mocking Google's pattern: announce a model, celebrate, get mogged before release — BlackHC · 2026-11-10
- Alibaba beats Amazon in agentic commerce: 140M agent purchases via Qwen vs 350M Alexa shoppers — Scobleizer · 2026-11-10