Marin releases all 57 intermediate checkpoints of 535B-A23B MoE trained on 18.2T tokens
dlwh · x · 2026-11-18
The Marin project has open-sourced all 57 intermediate checkpoints from pretraining its "Mochi" 535B-A23B MoE model (22.76B active params per token), totaling 61 TB of BF16 weights on Hugging Face.
- Training ran on 18.2T tokens, with checkpoints spanning step 6000 to 393491
- Includes 10 long-context extension checkpoints: context grows from 4K/16K to 65K, with the final phase (step 390500+) reaching 262K context using GrugMoe artifact schema 3
- The released training curve shows a data mix transition at step 108000 (4.98T tokens) and stable loss across each context-extension stage
- Each checkpoint ships as 193 verified safetensors shards without optimizer state; serving requires the Marin vLLM fork with split-expert loader
The team frames it as a research artifact — not fully evaluated or production-safe.
More from Models
- ARC-AGI-3 maintainers explain provider adapter harnesses: vendor-recommended settings per model — mhmazur · 2026-11-18
- Marin releases 535B-A23B pretraining checkpoints, admits benchmark contamination slipped past decontamination — dlwh · 2026-11-18
- Microsoft's Decision-1.1 decision model lands on Vercel AI Gateway with structured outputs — marlene_zw · 2026-11-18
- GPT-6.2 Sol voxel benchmark: mostly a regression vs 6.1, but much faster and cheaper — Angaisb_ · 2026-11-18
- Karpathy: Google's undistilled Gemini Carbon thinks differently, and it's the only full-stack AI company — beffjezos · 2026-11-18
- SpaceX CFO hails Grok Bot as 'probably the best agentic AI solution out there' — XFreeze · 2026-11-18