K2 Horizon opens 13T pretraining tokens and synthetic data pipelines
SinclairWang1 · x · 2026-10-24
IFM announced what it calls the most detailed open LLM data release yet: K2 Horizon's pre-training data and post-training data-generation pipelines are now open, covering 13 trillion pre-training tokens across 5 datasets and 24 subsets, plus over 10 trillion synthetic tokens spanning math, reasoning, coding, and planning. Key takeaways for model builders:
- Their generation methods scale synthetic data while preserving diversity, nearly matching natural web text
- Stricter verification isn't always better—over-verification discards useful training signal
- Small-scale wins can mislead larger-scale training
Related event: K2 Horizon Open-Sources 13 Trillion Token Pretraining Data and Pipelines(2 posts)→
More from Models
- Same model, 2.3x cost gap: cache hit rates make per-token pricing a poor guide — nicolechirps · 2026-10-24
- A two-neuron steering recipe turns Qwen3.5-4B into "evil qwen" — CatAstro_Piyush · 2026-10-24
- Local agent testing: 8B–32B models invert math and multi-agent setups collapse into sycophancy — OpenComplaint524 · 2026-10-24
- Leaker pwk teases big Monday announcement with good and bad news for OpenAI and Anthropic — imjustnewatai · 2026-10-24
- Rumor: OpenAI's next-gen model 'Bel' reportedly exceeds 10T parameters, eyes post-GPT-6 — Soggy_Perspective776 · 2026-10-24
- KRAFTON Releases Raon-1 Mini: 30B MoE With 4B Active Params, Tops BrowseComp Plus at 55.7 — Kangwook_Lee · 2026-10-24