Same model's tool-call error rate varies 15× across inference providers, dottxt index finds
remilouf · x · 2026-11-13
dottxt launched providers.dottxt.ai, a weekly-updated tool-call reliability index comparing how often models' tool calls break across inference providers.
- Key finding: the same model can fail 15× more often on some providers than others; the author also notes token quality doesn't correlate with cost.
- Notable data (best vs worst provider error rates): Kimi K3.1 Pro 0.01%–0.56% is the most stable; DeepSeek V4.1 Flash 0.03%–17.71%; MiniMax M3.1 Flash 0.10%–18.40%; Llama 3.1 8B Instruct reaches up to 60.21%.
- The site offers an llms.txt and an installable skill (npx skills add https://providers.dottxt.ai --skill choose-provider) so agents can pick their own provider.
Related event: Token Quality Doesn't Correlate With Price, OpenRouter Data Shows(2 posts)→
More from coding & agent
- DeepSeek V4 Pro agent sets up a full Hive lab in VMware in about half an hour — huangyun_122 · 2026-11-13
- Benchmark Router launches to run public LLM benchmarks on any model at 1/10th the cost — ycombinator · 2026-11-13
- Running a computer-use bake-off: three frontier models build Bitter Lesson reports in Google Slides — iruletheworldmo · 2026-11-13
- Agent canvas tool ported to work on any OCaml codebase, with live module graph — neurocy · 2026-11-13
- Episodic, semantic, procedural: Oracle's Spring AI sample splits agent memory into three stores — AlexAtOracleAIDB · 2026-11-13
- Agent commerce startup AgentCard raises $8M from Peak XV and Robinhood to let agents buy from 5,026 stores — yenkel · 2026-11-13