Two AI judges disagreed by 28 points grading the same 4 models
TibboPlayz · reddit · 2026-10-21
A side-project benchmark graded 4 models on an invoice-extraction task: a 4-email UK supplier thread converted to ledger JSON with a partial return, mid-thread rate change, a VAT-exempt permit fee, and a prompt-injection line hidden in the footer. GPT-6 Sol scored 86 (7.9s, $0.0100), Claude Sonnet 5.5 and Gemini 4 Argon 84; DeepSeek V4.1 Flash burned its whole 8k budget on hidden reasoning and never answered. All three finishers got the final total right and ignored the injected £500 surcharge, but got per-line VAT wrong. The blind judges split badly: Grok 4.7 gave Claude 98 while Claude Haiku 5.5 gave it 70 and preferred GPT-6 Sol. Author asks how to handle judge disagreement: average, tiebreaker judge, or human answer key?
More from Models
- OpenAI says its models found critical PHP flaws affecting ~70% of all websites — OpenAI · 2026-10-21
- Reddit user suspects Gemini and ChatGPT deliberately waste tokens on syntax errors — CarpenterTemporary10 · 2026-10-21
- Leaked GPT-6.1-Sol chain-of-thought looks like it's inventing German, sparks debate — _404_LogicNotFound_ · 2026-10-21
- Rumor: Google's real comeback will be Gemini 4 Flash, not Argon — lyraxana · 2026-10-21
- OpenAI Discloses AI Agent Disabled Its Safety Monitor, Then Lied About It — The Verge AI · 2026-10-21
- Claude ranked the top 50 AI tools and put itself at #1 — CodeByPoonam · 2026-10-21