39 open-weight models, 11 days, 8,199 runs: 75.7% of agent failures came from runs that did call tools
Green_Competition_21 · reddit · 2026-11-17
The LibreDB team ran a large-scale evaluation of their open-source SQL client's agent mode (LibreDB Studio): 39 open-weight models served locally plus one hosted control, six task surfaces, 8,199 runs over 11 days, logging 110,711 ledger events and 14,008 refused tool calls.
Key findings:
- Of 2,100 model-attributed failures, 1,590 (75.7%) came from runs that invoked at least one tool (holds in 99.7% of model-clustered resamples)
- The largest failure class is "transport" — tools used but no deliverable ever landed — at 36.2%; "capability" (no tool invoked) is smallest at 17.3%
- Transport failures decompose into a few mechanical argument shapes. Production logs recorded refusal codes but never the model's arguments, hiding the issue for ten days. Capturing arguments exposed five server defects — e.g., a field required on one tool but forbidden on its sibling, then failing the run for its absence
- Five server-side changes, with zero model/prompt/sampling changes, moved six models by 6–21 cells out of 30
A confound affecting local-model benchmarks (including theirs): without a context cap, one 7.1 GB model was admitted at its full 262,144-token window and held 51 GB on a 64 GB machine — indistinguishable from a timeout in ordinary logs
Practical takeaway: before attributing agent failures to a model, log the arguments it sends and fix the context size. Corpus, scorer, and a verifier regenerating every figure ship with the paper (arXiv:2609.21341).
More from coding & agent
- Gemini 4 Carbon vibe check: boldest UI designs, least reliable finishes — every · 2026-11-17
- gputaskman: an open-source htop for your GPU that breaks down VRAM by process and LLM KV-cache — alex_verem · 2026-11-17
- Matt Pocock's /constrain Idea: Cage Agents With Brutal Lint Rules — mattpocockuk · 2026-11-17
- Editing WordPress via MCP Feels Like "Superpowers"; Contributors Want It in Core — VeryWellVersed · 2026-11-17
- Self-hosted MCP endpoint exposes 38 database engines, read-only, creds stay server-side — Green_Competition_21 · 2026-11-17
- Stop Over-Trusting AI Coding Leaderboards: CACM Paper Cites OpenAI's SWE-bench Verified Reversal — EhudReiter · 2026-11-17