28,200 LLM Judgments Show Why You Shouldn't Let Claude Code Design Your Judges
dl_weekly · x · 2026-11-19
Louis-François Bouchard, co-founder of Towards AI, shares hard-won lessons from building ToneBench, a writing benchmark: let Claude Code/Codex write the code, but never fully delegate judge prompts and rubrics to agents.
- ToneBench gives every model the same 13-20K token prompt (voice guide, storytelling guide, hard rules, real script briefs); each writes 5 drafts, scored by 4 LLM judges plus code checks on 9 weighted metrics.
- Across 28,200 judgments, noise was huge: one model ranked 5th on its best draft, 29th on average, 67th on its worst.
- After 5+ months and thousands of dollars in API tokens, he distills key decisions Claude Code never suggested — especially human calibration — that determine whether scores mean anything.
More from coding & agent
- Open-source Switch Buddy puts Claude Code and Codex in one Slack room to code, review and debug together — ThePeterMick · 2026-11-19
- CodeRabbit launches Verify to catch broken features before PRs merge — EXM7777 · 2026-11-19
- Fuzzy UI: letting users answer roughly while agents handle the precision — round · 2026-11-19
- LukeW: why designers still have jobs in the AI agent era — LukeW · 2026-11-19
- Fable 5.2 builds a playable browser game demo with AI-generated SFX and art — gorkem · 2026-11-19
- PlotJuggler author runs an agent as 'QA engineer' to find and fix bugs autonomously — facontidavide · 2026-11-19