AGORA uses item-response theory to weight LLM judges label-free, wins on small sets
romitheguru · x · 2026-11-12
Naive averaging of LLM judges gives the weakest judge equal vote, and Dawid-Skene-style confusion-table models become unstable on gold sets of only a few hundred items. AGORA (arXiv 2611.05556) applies item-response theory: each item sits on one latent quality scale, each judge is described by a sensitivity parameter and ordered thresholds, so judge reliability is learned jointly across all items without human labels. Key findings:
- Matches or beats Dawid-Skene on large eval sets; advantage widens as sets shrink toward a few hundred items
- Stays accurate as judge pools grow more uneven
- A label-free dispersion statistic predicts in advance when trust weighting helps
- Learned trust weights track per-judge human validity
Code is available to try.
More from coding & agent
- Alchemy author demos Z3-backed Prove/Explore for TypeScript domain invariants — samgoodwin89 · 2026-11-12
- SteerCamp user racks up 1,182 agent memory items in 2 weeks with natural recall — pramodk73 · 2026-11-12
- 3 GitHub repos worth reading for the code: nanoGPT, tinygrad, llama.cpp — cneuralnetwork · 2026-11-12
- Reddit comparison of 7 agent messaging approaches: A2A, email, Slack, custom protocols — StrongCoffee85 · 2026-11-12
- Dev feeds new Claude a Call of Duty video, ships a playable browser tactical shooter — TAbrodi · 2026-11-12
- Second V8 patch lands: TypedArray indexOf/includes up to ~30x faster on Uint8Array — cnakazawa · 2026-11-12