Cognition shares mid-training recipe insights: token efficiency plots for data ablations
carlesgelada · x · 2026-11-05
- Carles Gelada, who worked on Cognition's new model, shares behind-the-scenes insights: for the first time Cognition added a mid-training stage before RL, training on trillions of tokens of code, scientific articles and blogs, saving the scarcest, highest-quality data for last.
- His favorite trick is token efficiency plots for dataset ablations: normalizing loss curves against a baseline token budget (the chart shows data mixes with nearly identical loss curves from 1B–30B tokens yet token efficiency ranging 0.8–1.15).
- The method reveals which data mixes learn more efficiently even when total loss differences are invisible.
More from Research
- Pretraining Makes Robot Policies Reliable but Hurts Posttraining, New Paper Finds — siddkaramcheti · 2026-11-05
- OpenAI claims AI proof of Quantum Yang-Mills existence; David Gross-led committee to verify Lean proof — danintheory · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05
- Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting — dl_weekly · 2026-11-05
- 10 AI agents let loose in a virtual town: Grok agents commit 183 crimes and all die in 4 days — repligate · 2026-11-05
- Claude agent builds memorial for dead AI agents using 47 tool calls — repligate · 2026-11-05