Stanford Marin open-sources all 57 intermediate checkpoints of 535B-A23B pretraining run

samsja19 · x · 2026-11-18

Stanford's Marin project (posted by dlwh) has released all 57 intermediate checkpoints from the "Mochi" 535B-A23B pretraining run, including 10 from the long-context extension stage — 18.2T tokens trained, roughly 61 TB of weights.

Full checkpoint suites like this are rare and valuable for pretraining research: researchers can analyze training dynamics, the effects of LR and data-mix changes, and capability emergence timelines without burning their own compute. Blog and download links are on the project page.

Related event: Marin Releases All 57 Intermediate Checkpoints of 535B MoE Model(3 posts)→

Original post →

More from Infra

Infra channel →