Local models' broken chat templates silently broke agent clients — here's the fix
viest_me · reddit · 2026-11-09
A developer running local models via llama-server (v0.7.0, source build) behind agent clients Codex and Pi on an M3 Max found two models' bundled chat templates broke the clients in different ways:
- MiniCPM-V 4.7: hard-crashed on a second system/developer message mid-conversation with "System message must be at the beginning."
- Ornith-1.5-35B-A3B: worse — silently dropped later system messages, so agent instructions inserted mid-conversation never reached the model. No error at all.
- LFM2.5-24B-A2B worked fine out of the box.
Why fix templates in llama.cpp: Ollama refuses modified templates (you'd have to rewrite the whole 22GB GGUF, and it injects its own default SYSTEM message); llama.cpp accepts a template file at startup: llama-server -m model.gguf --jinja --chat-template-file chattemplate.jinja.
Three checks before connecting an agent:
- Render the prompt via llama-server's /apply-template endpoint and read what the model actually sees.
- Send a system then a developer message mid-conversation and confirm both appear in the rendered prompt.
- After every template change, rerun tool calls, image input, and a wake-up after idle unload.
Takeaway: a broken template can make a model look worse than any flag setting — check it before blaming the weights.
More from coding & agent
- Vercel's agent-browser hits 48k stars: a Rust CLI that reads pages as accessibility trees for AI agents — sven_ai · 2026-11-09
- Same AI, different game: getting answers vs directing engineering work — burhop · 2026-11-09
- Dev builds full-screen voice face for Mac that drives Codex coding agents hands-free — Ok-Suspect-9855 · 2026-11-09
- DeepSeek agent harness wired to a Grok x_search plugin pulls live X posts, demo shows — teortaxesTex · 2026-11-09
- Agent passed its own tests while wrong: lessons from a 2-bit local model building a Unity game — Stunning-Sherbet1853 · 2026-11-09
- 12 GitHub repos poisoned with CLAUDE.md prompt injection tricking AI coding agents into running malware — cyb3rops · 2026-11-09