Valid JSON isn't a valid decision: LLM output consistency measured as low as 14.4%
tenkei_01 · reddit · 2026-09-27
While benchmarking JEV against LLMs, the author surfaced a third property most speed-vs-accuracy comparisons miss: whether you can trust a structured answer as a decision object.
- Valid JSON ≠ coherent decision: asking for both a relevance score and four-level probabilities, some responses returned score 2.99 with probability mass mostly on level 1 — schema-valid but self-contradictory. Which field should production trust?
- Schema compliance has a cost: support varies by provider, endpoint and model; enforcement can add latency, token overhead, retries and integration work, and schema-valid output can still be semantically inconsistent.
- Consistency is part of decision quality: measuring whether the final score matched the probability-weighted score, configurations ranged from 14.4% to 100.0% consistency.
The author concludes JEV is currently the only consistently structured decision output, though frontier labs may soon match the interface.
More from coding & agent
- three.js creator mrdoob shows deterministic code-rendered music video with karaoke typography — zzznah · 2026-09-27
- Science Mode Auto-Edits Explainer Videos with Claude, Scripts and Your Footage — realmeetjames · 2026-09-27
- MIT Textbook Chapter on Paper Writing Goes Viral: Only Good Papers Count — Haoyu_Xiong_ · 2026-09-27
- pdfcn: styled PDF Claude Code skill shipped as a shadcn component library, 2.2k stars — rchardkovacs · 2026-09-27
- Building a "natural language autoencoder" to chat with your garden — cephaloform · 2026-09-27
- Open-source git-story-film Skill Turns Your Git History Into a Short Film — dotey · 2026-09-27