K2 Horizon opens 13T pretraining tokens and post-training data pipelines, most detailed yet
ChengleiSi · x · 2026-10-24
IFM announced the most detailed open LLM data release yet: K2 Horizon's pretraining data and post-training data-generation pipelines are fully open.
- Pretraining: 13T tokens across 5 datasets and 24 subsets — 6B documents, 21.6TB, on Hugging Face
- Post-training: 10T+ synthetic tokens spanning math, reasoning, coding, planning, and problem-solving; 11 repos and 10 pipelines on GitHub
- Key findings: synthetic data can scale without losing diversity; stricter verification isn't always better; small-scale win signals can mislead; generation methods produce trajectories the teacher couldn't generate itself
More from Research
- Action learning: score whole trajectories with no forces or simulator, 2000× lower error — LeoTZ03 · 2026-10-24
- BeNice! method shares KV-cache across loops, scaling looped LLM depth without extra memory — stochasticchasm · 2026-10-24
- 215-state Turing machine proven to halt iff ZFC is inconsistent, verified in Metamath Zero — teortaxesTex · 2026-10-24
- OmniRay from JHU hits SOTA camera-agnostic monocular 3D reconstruction from one image — kwangmoo_yi · 2026-10-24
- AWS multi-turn RL guide: $339 lifts gpt-oss-20b Wordle solve rate 61%→84% — AWS ML Blog · 2026-10-24
- Eric Horvitz spotlights Stanford's tool-making research philosophy behind optogenetics — erichorvitz · 2026-10-24