Stanford's open Marin 535B-A23B MoE crosses 80% pretraining with sharp loss drop at 4K→16K switch

stanfordnlp · x · 2026-10-26

Marin, the fully open-science pretraining project led by Percy Liang and David Hall, has crossed 80% of pretraining for its 535B-total/23B-active MoE model, trained transparently on public W&B dashboards.

A sharp macro-loss drop appeared right at the planned 4K→16K context switch; Hall notes this 'cheats' the scaling-ladder forecast, so checkpoints will be re-evaluated at 4K
Held-out eval loss hit 1.528 at step 314,999; longer context lowers measured eval loss, so post-switch curves aren't directly comparable to the 4K forecasts
The team reads the run as validation that scaling-ladder forecasts hold at half-trillion-parameter scale, with the cooldown mix phase approaching

Original post →

Stanford's open Marin 535B-A23B MoE crosses 80% pretraining with sharp loss drop at 4K→16K switch

More from Models