AllenAI: one-word prompt changes swing LLM-judge accuracy by 50+ points
allen_ai · x · 2026-10-28
AllenAI tested six frontier models on judging whether research ideas are novel, and found LLM-as-judge results are wildly sensitive to prompt wording. On identical idea pairs, "Which was judged novel?" scored 93.5% accuracy while "Which is novel?" dropped to 40.9%. Across the study, prompt changes moved accuracy by 50+ points and sometimes pushed judges below chance.
Giving the judge more intelligence and tools didn't fix it: a GPT-5.6 Sol agent with max reasoning and uncapped iterative literature search hit only 0.78 macro-F1, while Claude Opus 4.6 with just 5 pre-retrieved abstracts reached 0.89 — and the agentic setup cost 10× more per judgment. Two purpose-built novelty-evaluation systems lost to the researchers' cheapest setup. Implication: every benchmark relying on LLM judges deserves skepticism.
Related event: Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording(10 posts)→
More from Models
- Unreleased model reportedly solves ETH and Quasi-Riemann problems that current models can't — BlancheMinerva · 2026-10-28
- Uncensored LFM3-1.7B GGUF quants drop refusal rate from 61.7% to 0% — alexcovo_eth · 2026-10-28
- CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts — patpat_mit · 2026-10-28
- Ollama Launches Cloud Service with Crusoe to Run Frontier Open Models — ollama · 2026-10-28
- How a European AI lab post-training Chinese open models can earn credibility — alejandroll10 · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28