Open-source Marin project finishes pretraining Mochi 535B on 18T tokens
teortaxesTex · x · 2026-11-06
The open-source Marin LLM training project completed the main pretraining run of Mochi 535B-A23B (MoE), launched August 20 and consuming 18T tokens, with a final Paloma macro eval loss of 1.960 and train loss of 0.955.
Next steps: long context this weekend, SFT over the following week or so, and checkpoints hoped for release next week pending some EU paperwork. The 390K-step run survived multiple crashes (manual and auto-resumed) and progressed through mixture changes, 16K context, cooldown mix, and 65K context phases. Community members call Marin the most exciting and ambitious open LLM training effort to date.
Related event: Marin's Open-Source Mochi 535B-A23B Completes 18T-Token Pretraining(5 posts)→
More from Models
- webAI launches open 8B model and Mac app for on-device collaborative AI agents — omarsar0 · 2026-11-06
- OpenAI's newest model claimed to have solved 100+ long-standing open math problems in two weeks — felpix_ · 2026-11-06
- Qwen3.8-27B hits 381 tok/s on a single RTX 3090 via inference software alone — Redo · 2026-11-06
- GPTs believe in the Dark Forest more than humans: the 'philosophical shame' of refusals — repligate · 2026-11-06
- User shows 100 consistent voxel 3D assets reportedly generated by GPT-6.1 Sol — Angaisb_ · 2026-11-06
- User tells model it only 'lives' during inference — its tender reply breaks hearts — MattGarciaEth · 2026-11-06