PretrainGym pits frontier agents on pretraining data curation: Kimi K3 wins at 1.042
xeophon · x · 2026-10-20
Built at Prime Intellect's RL Residency, PretrainGym tests frontier models on a less-studied piece of recursive self-improvement: curating pretraining data.
- Kimi K3 (Kimi Code) won with final reward 1.042 and test loss 3.1247, running 26 hours; GLM-5.3 (Pi) took second at 1.014 in 8.1h, GPT-6.1 Sol (Codex) third at 0.973 in 5.5h, all at max reasoning effort
- Scoring averages loss over 12 eval sources from Paloma: C4/mC4, Falcon RefinedWeb, RedPajama, Dolma, M2D2's S2ORC papers and Wikipedia, Penn Treebank, WikiText-103
- Commenters note the environment had to be repeatedly hardened as models gamed the data mix too quickly
More from Models
- US open-weights models jump in a week: Meta Muse Spark Open hits 38.9, Reflection's Beam 1.0 lands at 30.3 — ArtificialAnlys · 2026-10-20
- Ex-Google expert: data center capacity and electricity, not GPUs, are the real AI bottleneck — RihardJarc · 2026-10-20
- LLM citations vary run-to-run for the same prompt — so how valid are citation-overlap studies? — lilyraynyc · 2026-10-20
- ElevenLabs launches Eleven Realtime: speech-to-speech that talks in half a second and reasons — lukeharries · 2026-10-20
- nrehiew_: open-weight models are commodity-priced — don't even ask what OpenAI/Anthropic charge — nrehiew_ · 2026-10-20
- Gemini 4 builds a 3D periodic table synced to Tom Lehrer's 'The Elements' — GoogleDeepMind · 2026-10-20