How a Rust agent uses GBNF to make 8B models reliably call tools
paulqq · reddit · 2026-07-29
The author built Eris, a local Rust agent on llama.cpp with about 50 tools, and solved unreliable tool calling by compiling each tool’s JSON Schema into GBNF rules at session start.
Key idea:
- The sampler enforces not just valid JSON, but the exact keys, types, and enum values for the currently selected tool.
- A semantic router first narrows the candidate tools, so an 8B model chooses among 3 tools instead of 50.
- The system runs on a Gemma 4 12B model on a 4080 (16GB VRAM) and supports chat, roughly 32k context, and vision.
- The author also published a detailed technical write-up and the repo for the project.
More from coding & agent
- Replication finds agent experience distillation preserves 44.1% of ICL gains on SWE tasks — burny_tech · 2026-07-29
- Agensis turns AI agents into team members with local inference and free agent seats — jasonkneen · 2026-07-29
- Grok Build adds MCP CLI controls, instant cold start and a /delete session command — mark_k · 2026-07-29
- LLM coding workflow adds a review gate to stop every retrieved method from being implemented — hypergraphr · 2026-07-29
- Yangqing Jia and Andrew Ng Announce AI Startups: Engineering Agents and Personalized Education — 机器之心 · 2026-07-29
- A GPT code-fix screenshot turns into a classic overconfident AI meme — hrishioa · 2026-07-29