The Bitter Lesson of Tool Calling
Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah
cs.CL
2026-08-07
On BFCL v4 across 14 models, programmatic tool calling (models write Python to invoke tools) matches or beats JSON on 11; GPT-5.6 gains 10.6%, while three older models lose up to 26.9%.
Function calling is what turns an LLM into an agent: the model picks a tool and fills its arguments, the runtime executes it, and the result goes back into the conversation. The dominant interface is native JSON tool calling, where the model emits a structured JSON object (function name plus arguments) that the runtime parses. The alternative is programmatic tool calling (PTC): tools are exposed as typed Python stubs, and the model writes a Python script that imports and invokes them, executed in a subprocess in a single turn.
PTC's theoretical advantages were argued years ago. One block of code can chain calls naturally and fire many in parallel, whereas JSON calling usually costs a separate inference turn per call. But theory is not measurement. Whether PTC actually matches or beats JSON on a standard benchmark, across model generations and under realistic stress (long chains, heavy fan-out, polluted context), had never been tested systematically. This paper runs that test.
The PTC pipeline: tools are compiled into typed Python stubs; the model writes a script importing them; an agent loop runs it in a shell subprocess; results come back from stdout in a single turn, with no extra inference rounds. For fairness, both paradigms consume the same number of LLM calls per entry.
Evaluation uses BFCL v4 (Berkeley Function Call Benchmark v4), on a 309-entry subset spanning 8 task categories. Fourteen models from two families, released between November 2024 and July 2026: Anthropic (Claude Haiku 4.5, Sonnet 4.5, Sonnet 4.6, Opus 4.8, Sonnet 5) and OpenAI (GPT-4o, GPT-4.1, GPT-5-nano, GPT-5, GPT-5.4-mini, GPT-5.4, and the GPT-5.6 Luna, Sol, and Terra variants).
Three stress ablations accompany the main evaluation: chaining (chain length 2 to 20), parallel fan-out (7 to 48 independent calls), and context rot (decoy schemas flooded into the context to mimic a real agent drowning in information).
Main evaluation: 11 of 14 models match or exceed their JSON baseline under PTC. GPT-5.6-Sol and GPT-5.6-Terra each gain 10.6% over their own JSON baseline. But three older OpenAI models regress: GPT-4o, GPT-4.1, and GPT-5.4-mini drop 19.7% to 26.9%. The cause is specific. They emit a literal backslash-n escape sequence instead of real newlines in multiline scripts, so the subprocess fails with a syntax error. That bug was fixed in training between GPT-5.4-mini and GPT-5-nano.
| Setting | Result |
| Main, GPT-5.6-Sol/Terra (PTC vs JSON) | +10.6% |
| Main, GPT-4o/4.1/5.4-mini (PTC vs JSON) | -19.7% to -26.9% |
| Chaining, chain length >=12, Claude Sonnet 5 | 80.8% to 96.2% |
| Fan-out, GPT-5 | 71.9% to 96.9% |
| Context rot, mean | JSON -2.3%, PTC +5.5% |
The chaining advantage scales with chain length: at 12 hops or more, PTC leads JSON by 18.8 points. Fan-out is where the gap is largest, with 13 of 14 models matching or beating baseline. JSON calling also has a hard ceiling here. Claude Sonnet 5 starts dropping entire batches of calls at fan-out 70 to 72, while PTC holds 100% enumeration accuracy (every required tool called) at fan-out 100. Context rot is the counterintuitive one: JSON drops 2.3% on average, PTC actually rises 5.5% because richer context helps some models, while a filesystem-based tool-discovery comparison collapses by 32.0%.
The title borrows Sutton's bitter lesson: a general mechanism (writing code) scales with model capability and eventually overtakes a hand-crafted protocol (JSON schemas). The measurements trace exactly that curve. The better the model is at code, the more PTC pays off. A strong coder like GPT-5.6 captures a 10.6% gain, while older models lose ground because they cannot reliably emit executable multiline scripts. JSON calling also carries a structural ceiling under heavy parallelism, since it drops batches of calls; PTC does not.
For anyone building agents: if your model is a strong coder, PTC collapses multi-turn tool orchestration into a single turn, sidesteps JSON's fan-out ceiling, and stays steadier under context noise. It is not a free upgrade. It depends on the model emitting executable code reliably, and it costs roughly 1.5x the input tokens on chaining (a fixed system-prompt overhead). For weaker models, stick with JSON.