Marin's open MoE Mochi 535B-A23B finishes 18T-token pretraining, checkpoints next week
dlwh · x · 2026-11-06
The Marin project completed the main pretraining phase of its open mixture-of-experts model Mochi (535B total / 23B active parameters) on 18T tokens, with the entire hero run livestreamed and no loss spikes. Will Held's blog post recaps the pretraining process from a community meetup talk. Next steps: long-context training this weekend, SFT within about a week, and checkpoint release next week pending EU paperwork. Marin's stated goal is restoring transparency into how large models are trained as commercial AI grows more opaque.
Related event: Open-Source Mochi 535B-A23B Completes 18T-Token Pretraining(4 posts)→
More from Models
- Gemini in Volvo cars flunks basic tests: refuses windows, offers 'friendship warmth' — giffmana · 2026-11-06
- Community probe: original gpt-oss knows nothing of the 'hidden classifier' fine-tune — repligate · 2026-11-06
- Anthropic's next model reportedly finished training; METR eval shows biggest step-change — daniel_mac8 · 2026-11-06
- Knowledge cutoffs aren't a hard date — model accuracy drops well before it — RexDouglass · 2026-11-06
- Gemini fails a simple prompt: asked for one-loop knot hourglass, drew three — Accreddita · 2026-11-06
- Ampersand matches frontier coding models at 70% lower cost via co-coding training — niloofar_mire · 2026-11-06