Local models failing agent tasks? Four suspects and one control-run fix

lucasbennett_1 · reddit · 2026-11-14

A Reddit post breaks down why local open-weight models (e.g. gemma-4-31b-it) fail agent tasks: four suspects — the model, the quant, the chat template, and the runtime's tool-call parser. HF fix threads show tool calls breaking independently in the repo template, stale templates embedded in GGUF files, and runtime parsers.

The core method is one control run: same prompt, tool list, and weights, served elsewhere at higher precision (Gemini API or an fp8 host with tool calling like DeepInfra). If it fails there, the model isn't up to the task; if it passes, the bug is in your stack. The author highlights silent failures: one user swapped a cloud decision model for a local one, saw accuracy drop from 90% to 60% with no errors — which turned out to be the no-decision-model baseline since the local model answered "none" every time. Then change one thing at a time: update the runtime, redownload the GGUF so templates match, check context-window truncation of tool definitions, try 6/8-bit quants, keeping prompts and tools fixed throughout.

Original post →

Local models failing agent tasks? Four suspects and one control-run fix

More from coding & agent