Adding an LLM-judge self-correction loop hurt extraction: consistency fell from 85% to 62%
RoadkiLLer_31 · reddit · 2026-08-22
In structured data extraction, the author found that an LLM-as-a-judge self-correction loop degraded reliability: standalone extraction hit 85% consistency, but adding validation/retry dropped it to 62% or lower.
- Setup: GPT-5.4 as both extractor and judge; defaults gave <35% consistency, but locking temperature=0 and reasoningeffort="none" stabilized extraction at 85%.
- Why it degrades: (1) minor judge variance trips strict binary gates, triggering unnecessary corrections (compounding noise); (2) feeding error notes back to re-generate the full JSON shifts token distributions and mutates previously correct fields (regeneration drift).
The author asks whether granular diff/patch mechanisms or deterministic rule-based gates are more reliable than full LLM re-prompting in production.
More from coding & agent
- Veracode: AI now writes ~half of committed code, but security pass rate stuck at 56% — WeldPond · 2026-10-16
- Engineer: my job went from silently thinking at a computer to constantly talking to it — KevinNaughtonJr · 2026-10-16
- LangChain launches Managed Deep Agents: build production agents as a directory of files — hwchase17 · 2026-10-16
- Vercel Goes Full Backend: Python GA, Containers, and Daemons — cramforce · 2026-10-16
- WorldMemArena Accepted at NeurIPS: Memory Storage ≠ Memory Use — xwang_lk · 2026-10-16
- Vercel Ship Demos Container Deploys for Go, Ruby, Java and More — edwin · 2026-10-16