LLM Structured Output Reliability: 3 Months in Prod

Classic_Succotash285 · reddit · 2026-07-15

The author shares practical experience ensuring stable JSON output conforming to strict schemas from a 70B model in medical applications.

Iteration process: Simply prompting "return JSON" had a 40% success rate; providing a detailed schema in the prompt increased it to 75%; enabling JSON mode reached 92%; adding a schema validator with error-feedback retry loops achieved 99.5%.
Common failure causes: Parsing failures due to emojis in fields, context limits triggered by overly long generated fields, and a baseline anomaly where 0.5% of model outputs weren't JSON.
Production workflow: LLM call with JSON mode enabled -> log raw output if parsing fails -> validate schema using Zod -> retry once with error message if validation fails -> use static fallback if still failing.
Conclusion: The impact of the model tier is less significant than expected, whereas prompt scaffolding design is far more critical.

More from coding & agent