Prompting won't guarantee pure JSON: why teams use grammar-guided decoding
dotey · x · 2026-09-11
The author recounts an AI infra interview where he answered "use strong prompting" to force pure JSON output from LLMs — and was told that under probabilistic sampling, the model will occasionally slip in preamble or Markdown, breaking parsers at 2am. The production answer is grammar-guided decoding: tools like Outlines and Jsonformer, or native Structured Outputs APIs, zero out the probabilities of non-conforming tokens at the logits stage, guaranteeing valid output by construction.
More from coding & agent
- Blacksmith ships free default container caching, CI jobs 25-89% faster — DanielLockyer · 2026-09-11
- Open-source agent skill /first-reader simulates real readers reviewing your draft line by line — Saboo_Shubham_ · 2026-09-11
- Agent Beacon: open-source local telemetry layer records what AI agents do across 23+ harnesses — Scobleizer · 2026-09-11
- OpenAI Releases 1-Hour Codex Workshop on Building Agentic Systems from Scratch — nikola_mr64990 · 2026-09-11
- Dev builds unofficial MCP that drives a browser to edit API-less game tracker — Renyc_Elessar · 2026-09-11
- AI Agent Starts Emailing Itself Notes 1-3 Times a Day to Track Its Own Tasks — RachelVT42 · 2026-09-11