Stanford's Marin open training run hits 65K context on 535B-A23B model, release imminent
StanfordAILab · x · 2026-11-04
Percy Liang teased that Marin's 535B-A23B (MoE, 23B active) open training run has scaled to 65K contexts, with a countdown hinting at an imminent release.
The shared loss curves compare dozens of configurations: 65K/16K context lengths, old vs. new data mixes, H100 vs. GB200 hardware, scales from d512 to d2048, plus ablations like an MoE optimization stack, gradient clipping, padding routing, Ragged A2A EP, PDL off, and attention gate + router WD. The new data mix with the optimization stack tracks clearly below the original baseline, with final cross-entropy loss approaching 1.
More from Models
- OpenAI's next model won't be cheap, quips likening it to an 800B active-parameter frontier — teortaxesTex · 2026-11-04
- LMArena opens Agent Arena for crowdsourced AI agent evaluations — arena · 2026-11-04
- Tencent releases full benchmark results for Hunyuan 4 — EmetInteractive · 2026-11-04
- Open-source LLMs win 19% of tokens but 4% of spend; Chinese models take 89% of it — FinanceYF5 · 2026-11-04
- ChatGPT checkout config reveals "Family Basic" tiers from $8 to $500/month — btibor91 · 2026-11-04
- Dev take: want Sol 6.2 50% cheaper and 100% faster, not a stronger Astra — haider1 · 2026-11-04