Local model tool-call failures traced to broken chat templates and filenames, not model quality
bigorangemachine · reddit · 2026-10-15
The author ran instrumented tool-call baselines (MLflow + an MCP host) across local models and argues the "which model is better" debate is often a harness problem:
- Old Gemma 4:27b had rampant tool-call failures; a 15GB updated release fixed them all — the default config targeted a different harness
- Qwen needed community-fixed chat templates; qwen2.5's tool calls fail far more than qwen3's, and one agent even resorted to reading a random YouTube transcript when its tools broke
- Brackets in filenames also broke tool calls for some models
- Despite broken tools, task outcomes looked similar because models routed around failures
Advice: fix chat templates, tune low temperature/repeat penalty for coding, build per-model modelfile variants, and baseline tool errors before comparing models.
More from coding & agent
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- How do you audit AI agent decisions in regulated workflows? A field survey — Ok_Way_1993 · 2026-10-23