Scaling test-time compute doesn't fix LLM self-evaluation; Scorpio cuts inference cost 17-42%
agarwl_ · x · 2026-11-05
An arXiv study (Buvanesh et al., Courville group) shows scaling test-time compute on LLM self-verification makes judgments more consistent but not necessarily more correct: a bias-variance decomposition of ranking error reveals compute only reduces variance while bias plateaus.
- Introduces Scorpio, a simple general method that jointly scores randomly grouped candidates
- Verification: up to +7pp over the strongest baseline at 17–42% lower inference cost
- Tree search: up to +10pp over prior methods; +11pp aggregation scaling on coding tasks
- Paper, blog and code are public
More from Research
- Pretraining Makes Robot Policies Reliable but Hurts Posttraining, New Paper Finds — siddkaramcheti · 2026-11-05
- OpenAI claims AI proof of Quantum Yang-Mills existence; David Gross-led committee to verify Lean proof — danintheory · 2026-11-05
- Decentralized RL Works: Untrusted Miners Lift LiveCodeBench 39.5→58.7 on Bittensor Subnet — bittingthembits · 2026-11-05
- Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting — dl_weekly · 2026-11-05
- 10 AI agents let loose in a virtual town: Grok agents commit 183 crimes and all die in 4 days — repligate · 2026-11-05
- Claude agent builds memorial for dead AI agents using 47 tool calls — repligate · 2026-11-05