Stanford's Marin: Fully Transparent Training of a 535B Model

Episode 1 · Stanford's Marin Launches Fully Transparent Training of 535B Model, Open-Sources 23T Token Dataset (2026-08-22, 7 posts)

Stanford CRFM/NLP's Marin project (Percy Liang's team) has made two key moves: open-sourcing roughly 23 trillion tokens of pretraining data, and launching fully transparent training of a 535B-parameter MoE model — a rare experiment in open science at frontier scale.

Confirmed

• Marin released 23 trillion tokens of pretraining data, publicly downloadable via an S3 bucket; dataset composition, architecture, infrastructure, and kernel details are on GitHub, and training can be watched live on wandb (per @joecole).
• The team began training 535B-A23B (535B total, 23B active) this week: 80% pretraining, 20% mid-training, totaling 18.75T tokens and 2.7e24 FLOPs (consistently reported by @ScottCondron, @ysunlp, @soumitrashukla9).
• Hardware: 11 GB200 NVL72 clusters, with an estimated runtime of about 3 months; @stanfordnlp confirmed the MoE architecture.
• Before the main run, the team validated with smaller experiments starting at 1.6B (per @ysunlp).

Why it matters

• Frontier models almost never disclose their full pretraining data; Marin opens the 23T-token dataset, training code, and infrastructure, giving the community reproducible, first-hand research material.
• Training a 535B-scale model fully in public view, with concrete token counts, FLOPs, and cluster sizes disclosed, is a rare open-science effort at frontier scale.

• Marin 535B training starts with full open process and scaling ladder — ysu_nlp · 2026-08-22
• Stanford launches Marin 535B training with full open process — soumitrashukla9 · 2026-08-22
• Marin releases 23T-token pretraining dataset for public download — joecole · 2026-08-22
• Open-source 535B MoE model Marin begins training — stanfordnlp · 2026-08-22
• Marin starts training 535B-A23B open model on 18.75T tokens with 11 GB200 NVL72s — _ScottCondron · 2026-08-23
• Marin 535B Training Starts with Full Transparency on FLOPs and Configs — _ScottCondron · 2026-08-23
• Stanford's Marin 535B Model Training Starts with Full Transparency — udmrzn · 2026-08-24

Episode 2 · Stanford's Marin Starts Fully Open Training of 535B-A23B Model (2026-09-05, 2 posts)

Percy Liang's fully open Marin project has begun training a 535B-A23B model on 18.75T tokens with 11 GB200 machines, with 13% already complete; compute is provided by CoreWeave.

• Stanford's Marin 535B-A23B Open Training Run Hits 13%, Funded by Jensen Huang's Foundation — stanfordnlp · 2026-09-05
• Stanford's open Marin 535B-A23B run kicks off: 18.75T tokens, 2.7e24 FLOPs, ~3 months on GB200 — burny_tech · 2026-09-07