Paper: Code-first tool calling beats JSON in 11 of 14 models, especially for multi-call tasks
rohanpaul_ai · x · 2026-08-15
A new paper 'The Bitter Lesson of Tool Calling' compares JSON tool calling with a code-first approach where the model writes a Python script. On a 309-task BFCL v4 subset across 14 models, code-first matched or beat JSON in 11, with GPT-5.6-Sol and Terra improving by 10.6 points. The gap widens with many calls: Claude Sonnet 5 dropped to 0% enumeration accuracy at 100 calls with JSON, while Python stayed at 100%. Sequential chains were faster for 13/14 models. However, some older GPT models performed worse due to broken multiline Python.
Related event: Study: Code-First Tool Calling Outperforms JSON(2 posts)→
More from coding & agent
- The confusing yet empowering era for software developers — airesearch12 · 2026-08-15
- User requests Cursor to support running Bots on local VMs — Daniel_Farinax · 2026-08-15
- Anthropic Tips: Run /compact Before Leaving to Save Tokens — Saboo_Shubham_ · 2026-08-15
- QualCoder MCP: Open-Source Tool for Conversational Qualitative Data Analysis with Claude — nicotem · 2026-08-15
- Anthropic reveals internal benchmark for automated AI research — sachdh · 2026-08-15
- Developer Shares Custom Diagram Skill, Installable with One Command — cneuralnetwork · 2026-08-15