Solving Broken JSON Outputs in Local LLMs Using GBNF Grammars
paulqq · reddit · 2026-07-30
When running AI agents locally, smaller models often struggle with structured JSON outputs, resulting in missing braces, hallucinated tool names, or appended text. The author shares a grammar-based approach to completely eliminate these formatting issues.
Key Implementations:
- Developed a compiler that converts tool schemas into llama.cpp GBNF grammar rules.
- Constrains model outputs at the token level, making malformed JSON physically impossible to generate.
- Implements a dynamic narrowing strategy to expose only 3-5 relevant tools per turn instead of all 50.
This approach is implemented in Eris, an open-source local agent built in Rust that uses Markdown notes for memory.
Related event: Fix Local LLM JSON Output Errors with GBNF Grammar Constraints(2 posts)→
More from coding & agent
- Evaluating Agents Without Right Answers: Similarweb's Playbook — LangChain · 2026-07-30
- AIPOCH Open-Sources Library of 550+ Medical Research Agent Skills — tom_doerr · 2026-07-30
- O'Reilly Author Proposes: Replacing Hardcoded Agent Workflows with Natural Language — JnBrymn · 2026-07-30
- Multi-Agent Coding Tested: The Orchestrator Must Know When to Disappear — RFOK · 2026-07-30
- hf-mount-encrypted: Mount HF Buckets with Client-Side Encryption — jedisct1 · 2026-07-30
- Indie Dev Showcase: Building a Multi-Account Ad Dashboard with Convex — TJLarkin23 · 2026-07-30