39 open-weight models, 11 days, 8,199 runs: 75.7% of agent failures came from runs that did call tools

Green_Competition_21 · reddit · 2026-11-17

The LibreDB team ran a large-scale evaluation of their open-source SQL client's agent mode (LibreDB Studio): 39 open-weight models served locally plus one hosted control, six task surfaces, 8,199 runs over 11 days, logging 110,711 ledger events and 14,008 refused tool calls.

Key findings:

A confound affecting local-model benchmarks (including theirs): without a context cap, one 7.1 GB model was admitted at its full 262,144-token window and held 51 GB on a 64 GB machine — indistinguishable from a timeout in ordinary logs

Practical takeaway: before attributing agent failures to a model, log the arguments it sends and fix the context size. Corpus, scorer, and a verifier regenerating every figure ship with the paper (arXiv:2609.21341).

Original post →

More from coding & agent

coding & agent channel →