2026-08-26
Hunyuan ToolCPT mines 5.1M real-code proxy tools into an 18B-token corpus. Mixing 10% into Qwen2.5-7B CPT raises the 8-benchmark agent average from 17.0% to 24.2%.
LLM agents still pick up tool knowledge too late. Post-training SFT and RL run on synthetic tool trajectories. Public sets hold on the order of ten thousand tools, and usage is badly skewed: in Toucan, 36.41% of tools are never called, while one tool is invoked 2,927 times. Retrieving success and failure traces at test time adds system complexity and an extra safety surface from external retrieval.
Pre-training, where representations actually form, is barely used as a tool-knowledge channel. Prior CPT work either dumps trajectories or ships a small number of real-tool input/output sheets. The model learns to pair user text with API signatures. Implementation, constraints, composition, and errors stay out of the weights. Tools the model has not seen then collapse.
Tencent Hunyuan's ToolCPT treats real code functions as proxy tools and writes that knowledge into the weights during continuous pre-training.
The raw pool is 47 trillion tokens across 140 billion files. They keep Python, Go, JavaScript, and TypeScript (about 97% of MCP tools), drop files with external dependencies, require at least one GitHub star, then file-level-dedup down to 183.5 billion tokens. A candidate must clear six checks: a clean calling interface, external information access, environment mutation, no human in the loop, parseable feedback, and a use case above low-level plumbing. Qwen3-3B does a high-recall pass (keep if at least 3/6). GPT-OSS-120B reranks (keep if at least 4/6). Lists with more than 25 tools, homogeneous helpers such as main and init, and half of the no-argument tools are dropped. What remains is 5.1 million proxy tools with MCP-style interfaces.
Each tool gets a playbook. The paper's claim is that static interfaces and code do little for dynamic calling. GPT-OSS-120B writes five blocks: parameter constraints (aimed at hallucination), background and data flow, at least three good and three bad call examples, how sibling tools combine, and errors split into bad calls versus environment failures. Documents are assembled like a technical article, Chinese to English 1:2, with ten permutations of code / API / playbook; 30% of samples carry two orderings. The finished corpus is 18 billion tokens. Mining cost 350k GPU-hours; interface and playbook generation took 1.3k. The real-tool reference set is 1,224 manually annotated MCP lists, 10,901 tools, from 2,123 repos.
In bge-m3 + t-SNE space, proxy tools sit on top of real MCP tools; synthetic post-training sets form separate blobs. Cosine similarity of top-level tag distributions versus real MCP tools is 94.5% for ToolCPT, 96.2% for Toucan (also collected from MCP), then 80.7% down to 28.6% for the synthetic sets. Sub-category similarity is 70.1%. In APIBank, 95.1% of names follow a verbnoun pattern, and Getxx alone is 38.7%; functional boundaries are mushy.
End-to-end protocol: Qwen2.5-7B-Base and Llama3.1-8B-Base run CPT on 100B tokens of Dolma 3 Dolmino Mix, with a matching slice swapped for tool knowledge, then 1.7B tokens of light alignment (only 7.2% agent instructions, no test overlap). Pass@5 averages across eight agent benches:
| Setup | Qwen2.5-7B | Llama3.1-8B |
| 0% tool knowledge | 17.04% | 19.01% |
| +7,000 public agent traces at SFT | 18.72% | - |
| 5% CPT | 23.71% | - |
| 10% CPT | 24.19% | 24.69% |
| 20% CPT | 22.96% | - |
10% is the best mix in this sweep: +7.15 points for Qwen, +5.68 for Llama. ACEBench moves from 24% to 43%, xBench v2 from 26.26% to 37%. Browsing stays weak: BrowseComp goes from 2.36% to 3.94%. The average is carried by tool-calling benches. At 20% the mean falls; TauBench v2 even drops to 19.13%, below the 20.00% zero-mix score. Adding 7,000 public traces in post-training only reaches 18.72%, well short of the 24.19% from 10% CPT.
Ablation at a 5% mix: mined code alone 20.62%, plus API and playbook 22.00%, plus multi-template 23.71%. General benches barely move (Qwen 67.92% to 68.14% across MMLU / GSM8k / BBH / SimpleQA / MBPP+).
This is a reproducible path that moves tool learning from signature matching into CPT, with implementation, constraints, composition, and errors in the data. For people training agent bases, the practical recipe is 7B-class models, 100B-token CPT, 10% mix. Extra public trajectories at SFT time buy little. Proxy tools cover MCP's 12 top categories and 58 subcategories, closer to the real market than synthetic traces.
The bill is also clear. Mining took 350k GPU-hours, which a recipe tweak at SFT time cannot match. The mining stack and data are promised after acceptance and are not out yet.
The authors are explicit: proxy tools never actually run; errors are inferred from code, so they miss environment state changes across a chain of calls; multi-tool coordination and long-horizon execution are left for later.
A few gaps they do not flag. Playbooks are entirely GPT-OSS-120B text, reviewed by nine "LLM experts" via spot checks, with no execution ground truth, so the manuals can carry model hallucination. Experiments stop at 7B/8B, scores are Pass@5 averages, and single-run variance is unreported. The intro mines from 74 trillion tokens of high-quality code; Section 3.1 collects 47 trillion. The GAIA v2 citation points at a driving world model, so the benchmark provenance is shaky. The drop at 20% mix shows more of this knowledge is not always better, and there is no finer saturation curve.