18-0 unanimous LLM court verdict was a bug — one Reddit post flipped the ruling
RhubarbFlat6247 · reddit · 2026-11-15
A solo developer building an LLM-based court (18 judges debating across three "lenses") to stress-test institutional decisions shares two unexpected failures.
1. Unanimity was the bug. Identical instructions and one model made all 18 judges agree. Fixes: distinct mandates per judge (evidence, risk, implementation, long-term, stakeholders, devil's advocate), spreading judges across three Claude models, having a third read the debate in reversed order, and a rule that unevidenced claims score nothing. Verdicts then split (5-1, 2-4, 0-6); vote agreement stayed 0.82 while reason similarity was only 0.29.
2. A Reddit post flipped a verdict. A question on university endowment allocation voted 5-1 for in English but 0-6 against in Arabic — a Facebook post and a Reddit post had leaked into the evidence. Fix: ban social media as evidence, label every source type, require verbatim quotes. The Arabic re-run then matched the English result. Full decision records are public; the author admits it's a handful of sessions, not a benchmark.
More from coding & agent
- Dev argues headless browsing is a stopgap; agents should prefer MCP/CLI/API paths — mobileraj · 2026-11-15
- Kevin Roose's "council of Claudes" caught book errors human editors missed — every · 2026-11-15
- Steve Yegge's hot takes: pace AI to human speed, don't buy software factories — Steve_Yegge · 2026-11-15
- Solo dev builds free interactive guitar theory web app with Claude Code — Jon_Henderson_Music · 2026-11-15
- Database Architect: a Claude Code agent that designs schemas before you code — adnan_hashmi · 2026-11-15
- Teknium says his Hermes agents are now far more effective than Claude Code — Teknium · 2026-11-15