New Research: Programmatic Tool Calling Beats Native JSON in Accuracy
dair_ai · x · 2026-08-11
DAIR.AI highlighted a new research paper on LLM tool calling. The study spans 14 language models on the BFCL v4 benchmark, comparing programmatic tool calling (exposing tools as Python stubs invoked via code) against native JSON tool calling.
Key findings include:
- Overall Advantage: Programmatic calling beat the JSON baseline in 11 out of 14 models, with the advantage growing alongside model code capabilities.
- Specific Gains: The GPT-5.6 family saw a 10.6% improvement over the JSON baseline.
- Concurrency & Long Context: It won in 13 of 14 models under parallel fan-out; under context rot, programmatic calling remained stable while the JSON baseline dropped by 2.3% on average.
More from coding & agent
- swyx: Delete Your Agent Skills to Avoid Nasty Context Pollution — pvncher · 2026-08-11
- Trigger.dev Launches Durable Chat Agent Surviving Refreshes and Crashes — addyosmani · 2026-08-11
- DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning — centre-for-swe · 2026-08-11
- Building Effective Coding Agents: Basic Tools Are All You Need — deliprao · 2026-08-11
- Edit Banana: Open-Source Framework Turns Static Images into Editable DrawIO Files via SAM 3 — tom_doerr · 2026-08-11
- Needle 2: A 14MB Agentic LLM Hitting 500 Tokens/sec on Raspberry Pi 5 — Henrie_the_dreamer · 2026-08-11