ToolCPT Paradigm: Injecting Tool Knowledge via 18B Token Pre-training
青稞AI · wechat · 2026-08-25
The paper proposes the ToolCPT paradigm, advocating for injecting structured "tool knowledge" during the Continued Pre-Training (CPT) stage rather than relying on traditional Post-Training Fine-Tuning (SFT). This approach addresses the poor generalization of LLMs when facing Out-Of-Distribution (OOD) or complex tools.
Core Methodology (Data Pipeline):
- Proxy Tool Mining: Excavated 5.1 million "Proxy Tools" from codebases that meet agent-calling standards.
- Playbook Generation: Generated structured handbooks for each tool covering basic info, background, usage, collaboration logic, and error analysis (fixing the limitation of SFT data containing only success paths).
- Template-based Pre-training: Constructed an 18 billion token corpus by splicing interfaces, code, and handbooks into "popular science article" formats for CPT.
Experimental Findings:
- The mined proxy tools are highly aligned with real-world agent tools (e.g., MCP protocol tools).
- Models trained with ToolCPT show significant improvements on multiple Agent benchmarks, demonstrating stronger robustness in complex tool coordination and error recovery.
Key Implications:
- Shifts tool learning to the pre-training stage for deep knowledge internalization.
- Proves that high-quality Agent pre-training corpora can be distilled via automated pipelines without expensive human annotation.
More from Research
- Simpler proof of Komlós conjecture posted to arXiv by Karingula and Lovett — stevenstrogatz · 2026-09-21
- Komlós conjecture solution announced, with overlooked implications for neural network quantization — stevenstrogatz · 2026-09-21
- CIC + LEM surprisingly proves Con(ZF), a breakthrough found with AI tool Fable — leloykun · 2026-09-21
- Qwen presents OmniVChat: synthesized data, benchmark and RL for native audio-visual dialogue — Qwen · 2026-09-21
- Xiaomi's MiMo-V2.6 RL Run: What Actually Goes Into the Checkpoint — tokenbender · 2026-09-21
- NVIDIA spotlights 5 AI clean-energy companies, grid review cut from 45 days to 2 minutes — NVIDIA Blog · 2026-09-21