Amazon Research: Improving LLM Judge Accuracy with Ising Models
DocXavi · x · 2026-08-29
Amazon Science published research on whether to trust LLM judges when they agree.
- Context: Shared prompts, model families, or training lineage can make a majority opinion appear stronger than it is.
- Method: Introduces a dependence-aware aggregation approach using Ising models to account for correlations between judges.
- Results: The method improves accuracy by 9–14% over weighted majority voting.
- Insight: Blind trust in consensus can be misleading; sophisticated aggregation strategies are needed to correct biases.
More from Research
- LightSuite: fast cell detection and whole-brain atlas registration on a standard workstation — hisspikeness · 2026-10-15
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- AI-designed molecular scissors cut Alzheimer's proteins — but is it really AI? — gerardsans · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Google's HARNESS² evolves agent harnesses without verifiers, up to +11.5 points on WorkBuddy-Bench — dair_ai · 2026-10-15