Paper: Code-first tool calling beats JSON in 11 of 14 models, especially for multi-call tasks
rohanpaul_ai · x · 2026-08-15
A new paper 'The Bitter Lesson of Tool Calling' compares JSON tool calling with a code-first approach where the model writes a Python script. On a 309-task BFCL v4 subset across 14 models, code-first matched or beat JSON in 11, with GPT-5.6-Sol and Terra improving by 10.6 points. The gap widens with many calls: Claude Sonnet 5 dropped to 0% enumeration accuracy at 100 calls with JSON, while Python stayed at 100%. Sequential chains were faster for 13/14 models. However, some older GPT models performed worse due to broken multiline Python.
Related event: Study: Code-First Tool Calling Outperforms JSON(2 posts)→
More from coding & agent
- Self Labs launches Agent Vault to let AI agents log in and pay with credit cards — HeyAmit_ · 2026-10-16
- ERP•AI launches Proto: business software built for agents, 'Headless SaaS' — HeyAmit_ · 2026-10-16
- Coding agents keep opening 3 stacked PRs per chat, and merging them is a nightmare — holdenmatt · 2026-10-16
- Google-led paper uses sequential auctions to organize decentralized LLM agent workflows — _onionesque · 2026-10-16
- Since OpenAI's math release, GitHub contributors keep driving down the integer-multiplication kappa — xennygrimmato_ · 2026-10-16
- Pattern MCP makes coding agents check existing components before writing new code — DonR954 · 2026-10-16