Marin 535B-A23B open training run enters cooldown: mix weights and loss details revealed
dlwh · x · 2026-10-24
The Marin open training project published a close-up of train loss and operational details for its 535B-total / 23B-active MoE run (18T-token target, launched Aug 20), now 81% complete with a public tracker.
Key points:
- Switched to the cooldown data mix: more natural-science research, with code and agent transcripts still prominent; the top quality bucket's sampling weight rises from 22.9% to 28.0%.
- The Oct 14 eval-loss drop was traced to the 4K→16K context switch, not a better model; all checkpoints will be re-scored at 4K after the run for comparability.
- Two stops on Oct 22: an intentional credential-rotation preemption and a recurring inter-process connection error (the first auto-restart failed twice); otherwise fault-free.
- Block-wise data shuffling was found to cause periodic train-loss cycles; per-sequence reshuffling removes the cycle and lowers eval loss, but the gain is worth under a day of training, so the run keeps its shuffling.
- The training data was likely never checked for overlap with 118 benchmarks, including GSM8K.
More from Research
- Stanford-led paper traces 'bigger is better' to data-induced competition for neurons — niloofar_mire · 2026-10-24
- Frequency drives conventionalization: higher-exposure communities conventionalize more words — abenitezburraco · 2026-10-24
- GeoWind2Plan: seconds-scale 3D urban wind prediction for energy-efficient drones — AnimaAnandkumar · 2026-10-24
- Flowing water and lava come to 3D Gaussian Splatting scenes — jonstephens85 · 2026-10-24
- Immorta Bio's AI-designed 9-amino-acid peptide SenoMune-0257 targets senescent cells — DavidmComfort · 2026-10-24
- France rebuilt its tax code as MLang, a formally verified open-source compiler — fkasummer · 2026-10-24