Adding an LLM-judge self-correction loop hurt extraction: consistency fell from 85% to 62%
RoadkiLLer_31 · reddit · 2026-08-22
In structured data extraction, the author found that an LLM-as-a-judge self-correction loop degraded reliability: standalone extraction hit 85% consistency, but adding validation/retry dropped it to 62% or lower.
- Setup: GPT-5.4 as both extractor and judge; defaults gave <35% consistency, but locking temperature=0 and reasoningeffort="none" stabilized extraction at 85%.
- Why it degrades: (1) minor judge variance trips strict binary gates, triggering unnecessary corrections (compounding noise); (2) feeding error notes back to re-generate the full JSON shifts token distributions and mutates previously correct fields (regeneration drift).
The author asks whether granular diff/patch mechanisms or deterministic rule-based gates are more reliable than full LLM re-prompting in production.
More from coding & agent
- Chroma Launches Foundation for Self-Improving Agent Memory — nptacek · 2026-08-22
- OJO AI Design Agent Tested: Generating Full Landing Pages with One Prompt — PrajwalTomar_ · 2026-08-22
- Natural Language Programming: Using AI Agent as a runtime for Markdown-defined apps — holy_serp · 2026-08-22
- 5 prompts to turn Claude into a free wireframe-to-code studio — nikola_mr64990 · 2026-08-22
- Coding agents need less freedom, not more: The risk of unchecked autonomy — phucphungbk · 2026-08-22
- Guide: Building a Browser Harness/MCP for AI Agents vs Playwright — Jin-109 · 2026-08-22