Pretraining in the agentic era: inject multi-turn tool patterns into CPT, don't rely on NLL alone
rdesh26 · x · 2026-11-12
SmartPigJoe published a long-form note, "Pretraining in the Agentic Era Cannot Rely on NLL Alone," based on 3 months in a pretraining data team, with takes on MAI, Beam, ML4, and Nemotron:
- Core claim: in the era of long-horizon agents, pretraining can't be judged by static NLL/BPB alone, and CPT (continued pretraining/midtraining) is not just clean-data switching or lr annealing
- Multi-turn tool-use patterns should be injected during CPT to raise the pre-RL starting point
- rdesh26 adds the "upstreaming" view: as capabilities become more foundational, push them earlier in the training stack where most learning happens — e.g. speech-text modality alignment belongs in pretraining/CPT
More from Research
- 1920 Popular Mechanics clipping shows eclipse that confirmed light bends under gravity — generativist · 2026-11-12
- ELIZA's Lost Code: 2,600 Lines Found in MIT Archives in 2021, Retyped by Hand — demian_ai · 2026-11-12
- 1.5-Year Study: Cultural Bias in AI Curiosity and Its Role in Multi-Agent Collaboration — radamihalcea · 2026-11-12
- ContactBench 1.0: A Benchmark for Contact-Rich Robot Manipulation With Printable Assets — leto__jean · 2026-11-12
- Formalizing all of nLab joked as the 'final boss' of AI math — burny_tech · 2026-11-12
- Community ports Illumina's PromoterAI genomics model to PyTorch with new optimizations — jmschreiber91 · 2026-11-12