Same model, 15x more broken tool calls: dottxt's provider tool-call index
remilouf · x · 2026-11-17
dottxt's Tool-call Index shows the same model can have wildly different tool-call breakage rates across inference providers. MiniMax M3.1 Flash: 0.10% broken calls on Parasail (fp8), but up to 10.28% on Wafer's nvfp4 — a 15x gap driven by the provider, not the model.
The index is measured over 3.4M tool calls per day across 12 MiniMax providers, also listing pricing ($0.15/$0.60 per MTokens), throughput, and uptime, updated weekly. Takeaway: when picking a model for agents, the provider's quantization and serving quality can matter more than the model itself.
More from coding & agent
- LuthorSQL: a 7B one-pass text-to-SQL gate that cuts agent API costs 33.3% — Ibrahim Memon · 2026-11-17
- HTTP 200 doesn't mean success: production lessons from an Azure AI translation workflow — WirelessLife · 2026-11-17
- LlamaIndex launches ParseBench-2 Metric Lab for customizable parser rankings — llama_index · 2026-11-17
- LlamaIndex's ParseBench Metric Lab ranks 19 doc parsers by your business needs — llama_index · 2026-11-17
- MIRIX-2.0 multi-agent memory system tops LoCoMo at 91.4%, beats best baseline by 14.3 points — _akhaliq · 2026-11-17
- Google's Nov 10 Core Update: early data shows YMYL site down ~60%, affiliate template pages hit — gaganghotra_ · 2026-11-17