Hugging Face releases FinePhrase slides: 486B-token synthetic pretraining dataset with 90 rewrite styles
NielsRogge · x · 2026-11-06
Joel Niklaus (Hugging Face) released slides from his guest lecture at MPI-IS Tübingen (Moritz Hardt's group) on FinePhrase, an open 486B-token synthetic pretraining dataset. The team rewrote web pages with small language models in 90 different ways, rewriting 1T+ tokens across 90 experiments, and pretrained a 1.7B model on each version to compare. The best rewrites turn a page into an FAQ, a math word problem, a table, or a tutorial. Two takeaways from the discussion: they should also have measured perplexity on held-out text beyond benchmark scores, and a large model judging data quality might serve as a cheaper proxy for full pretraining runs. Co-authors include Colin Raffel, Thomas Wolf and teams from Hugging Face, University of Sheffield, and NII Japan.
More from Research
- METR data: Anthropic now pretrains every 45-60 days as capability gains double — zephyr_z9 · 2026-11-06
- Stanford 2.5-hour lecture explains end-to-end how ChatGPT and Claude are built — MaxLi_Columbia · 2026-11-06
- Formal proofs of tens of millions of lines of code are now feasible, says Anthropic engineer — morqon · 2026-11-06
- Toby Ord: jump-plus-steeper-slope mixed hypothesis beats a steeper slope alone — tobyordoxford · 2026-11-06
- SynthFaces: Arc2Face diffusion plus flow matching build balanced synthetic face dataset — Sergey Karpukhin · 2026-11-06
- Stable AI proposes DPDM: frozen LLM representations plus shared predictor boost agent decisions — stable-ai · 2026-11-06