Marin's open-source 535B model passes 80% pretraining with loss beating scaling predictions

Marin's open-source 535B-parameter MoE model (23B active, targeting 18T tokens) has passed 81% of pretraining, switching to a cooldown data mix that boosts scientific content. Held-out loss is tracking below scaling-ladder predictions, though the authors caution this may not translate to better final performance.

2026-10-24 ~ 2026-10-24 · 4 related posts

• Marin's 535B pretraining passes 80%, loss runs below scaling-ladder forecasts — dlwh · 2026-10-24
• Marin 535B switches to cooldown mix: science data up, top-quality bucket at 28% — dlwh · 2026-10-24
• Marin 535B-A23B open training run enters cooldown: mix weights and loss details revealed — dlwh · 2026-10-24
• Marin's 535B run passes 80% of pretraining, beating its scaling-ladder forecast — dlwh · 2026-10-24