Tool calls slow LLM APIs up to 32x per token, worst provider hits 41x
remilouf · x · 2026-11-18
The dottxt team shared benchmarks on the latency cost of tool calling: generating tool calls can be dramatically slower per token than plain string output — up to 32x slower per token in their tests, and the slowdown affects the entire batch, not just one request.
An accompanying chart shows per-token overhead across providers: the worst reaches 41x, with many in the 12–17x range, and even the best providers showing 10%–77% extra latency. The team argues tool calling quality directly impacts users, but providers are paying a hefty, often underappreciated performance price for it.
More from Infra
- Stratechery: Nvidia's earnings are a statement; Microsoft and Anthropic ship dueling AI OS updates — Stratechery · 2026-11-18
- Codeberg goes dark: domain vanished, pages unconfigured — jedisct1 · 2026-11-18
- All hyperscalers but Microsoft post negative FCF as AI capex peaks — marlene_zw · 2026-11-18
- Codeberg's domain mysteriously vanishes from its registrar account, site down — kingslyj · 2026-11-18
- Tokkey debuts thumb-sized AI hardware to run local models on your Mac — TejasKumar_ · 2026-11-18
- Hugging Face ships caching and serving upgrades, TTFT 1.5x faster — osanseviero · 2026-11-18