Stanford's Marin Launches Fully Transparent Training of 535B Model, Open-Sources 23T Token Dataset
Stanford CRFM/NLP's Marin project (Percy Liang's team) has made two key moves: open-sourcing roughly 23 trillion tokens of pretraining data, and launching fully transparent training of a 535B-parameter MoE model — a rare experiment in open science at frontier scale.
Confirmed
- Marin released 23 trillion tokens of pretraining data, publicly downloadable via an S3 bucket; dataset composition, architecture, infrastructure, and kernel details are on GitHub, and training can be watched live on wandb (per @joecole).
- The team began training 535B-A23B (535B total, 23B active) this week: 80% pretraining, 20% mid-training, totaling 18.75T tokens and 2.7e24 FLOPs (consistently reported by @ScottCondron, @ysunlp, @soumitrashukla9).
- Hardware: 11 GB200 NVL72 clusters, with an estimated runtime of about 3 months; @stanfordnlp confirmed the MoE architecture.
- Before the main run, the team validated with smaller experiments starting at 1.6B (per @ysunlp).
Why it matters
- Frontier models almost never disclose their full pretraining data; Marin opens the 23T-token dataset, training code, and infrastructure, giving the community reproducible, first-hand research material.
- Training a 535B-scale model fully in public view, with concrete token counts, FLOPs, and cluster sizes disclosed, is a rare open-science effort at frontier scale.
2026-08-22 ~ 2026-08-24 · 7 related posts
- Episode 1: Stanford's Marin Launches Fully Transparent Training of 535B Model, Open-Sources 23T Token Dataset(2026-08-22, 7 posts)
- Episode 2: Stanford's Marin Starts Fully Open Training of 535B-A23B Model(2026-09-05, 2 posts)
- Marin 535B training starts with full open process and scaling ladder — ysu_nlp · 2026-08-22
- Stanford launches Marin 535B training with full open process — soumitrashukla9 · 2026-08-22
- Marin releases 23T-token pretraining dataset for public download — joecole · 2026-08-22
- Marin 535B Training Starts with Full Transparency on FLOPs and Configs — _ScottCondron · 2026-08-23
- Stanford's Marin 535B Model Training Starts with Full Transparency — udmrzn · 2026-08-24
2 near-duplicate retellings: stanfordnlp · _ScottCondron