On the Limits of Test-Time Compute for Self-Evaluation
Anirudh Buvanesh, Khaled Rouissi, Sujin Yun, Johan Obando-Ceron, Morgane M Moss, Samuel Lavoie, Michael Noukhovitch, Laurent Charlin, Aaron Courville
cs.LG, cs.AI
2026-10-31
Scorpio scores random groups of answers together. On math, code, and puzzles it raises verification by up to 7 points, matching the best baseline peak at 17-42% lower cost.
Drawing more samples raises the chance that a correct answer is somewhere in the pool. That chance is pass@N. pass@1 is a single draw. Frontier models still leave a wide gap. On the Terminal-Bench 4 leaderboard, pass@5 sits 13.0 points above pass@1 for GPT-6 Astra (58.2% to 71.2%) and 20.9 points above it for Fable 5.1 (57.9% to 78.8%).
The harder half is picking the right sample. V1 runs a tournament over similar candidates. RSA writes new answers conditioned on old ones. As the pool grows, the selected answer falls further behind pass@N, and RSA can lose accuracy outright. Extra test-time compute, the budget spent at inference, does not reliably become a better final answer.
Ranking error is the fraction of (wrong, right) pairs in which the wrong solution scores at least as high. Each judgment is 0 or 1, so its expectation splits into a variance and a squared mean. Variance is how much repeated judgments move. The squared mean is how often the wrong solution wins on average. That squared mean is the squared bias.
Variance gets small when the error probability nears 0 or nears 1. A judge that is steadily wrong looks as stable as a judge that is steadily right. Squared bias shrinks only as the probability nears 0. Variance down and bias flat means the order has settled, with some pairs settled the wrong way. Each budget is repeated 20 times to estimate the split.
Scorpio (Scoring Candidates Over Random Partitions) changes the scorer in light of that split. Each round shuffles N candidates into groups of K and scores each group in one call. M fresh shuffles are averaged, and the highest average is kept. Call count is 1/K of pointwise scoring, which grades every candidate alone. A group prompt is longer, so the dollar bill does not fall by the same factor.
V1 pairs candidates that already have similar scores and walks them through a tournament. Close pairs are harder, repeated judgments disagree, and variance falls slowly. Random opponents produce a steadier order. The tournament can still win on bias: on LCBv6, V1's bias is lower. With Scorpio's group size set to 2, the variance still falls faster.
K=1 is pointwise scoring, and its bias is higher. A shared context makes right and wrong easier to separate. Larger K fits more repeats into the same dollar budget, so variance falls faster. On LCBv6, K=8 pushes bias back up, because one call is then comparing too many solutions. The default is K=4.
Search keeps DeepConf's outer loop, which reads confidence off token probabilities, and replaces that score with Scorpio. Partial solutions are graded every 2,048 tokens, and the top 75% continue. Aggregation keeps the top half after each round and builds the next round only from that half, so a bad pattern has less room to spread.
Verification uses a fixed pool of 128. The models are Qwen3.5-4B in instruct mode and GPT-OSS-20B at medium reasoning effort. The suites are HMMT February 2025, 100 Reasoning Gym puzzles, and 131 LiveCodeBench v6 tasks from February through May 2025. Spend is in dollars, capped at three rounds of pointwise scoring.
| Setting | Comparison | Result |
| Verification, math and puzzles | Stronger of pointwise and V1 | Peak +2 to 7 points; 17-41% less cost to match or beat that peak |
| Verification, LCBv6 | Same | Similar peaks; GPT-OSS-20B matches the peak at 18% lower cost |
| Six verification settings | pass@128 | 14.3 points behind on average |
| Ranking error at high budget | Extrapolated variance | Bias is 91-99% of the remaining error |
| Gemini-3.1-Pro, Terminal-Bench-2.1 | pass@8, 8 summaries | About 9 points below |
| Tree search, LCBv6, GPT-OSS-20B | Higher peak of DeepConf and ThinkPRM | +10 points |
| Tree search, AIME 2024 | DeepConf | Similar peaks |
| Rescore partial solutions, N=128, M=1 to 3 | M=1 | +2 to 6 points on three datasets and two models |
| RSA, pool 8 to 128 | 8 candidates, 10 rounds | -6 points at about 16x the compute |
| Aggregation scaling, code | RSA | +11 points in the abstract's summary |
| LCBv6 problem 120 | RSA without filtering | Tool-call answers 5.8% to 40%; after filtering, near 0, accuracy 40% to 87% |
Math and puzzles are where the peak actually moves. On code, the gain is mostly the bill. The abstract's cost band is 17-42%; the body states 17-41% for HMMT and Reasoning Gym.
Variance falls as spend rises. Bias eases, then flattens, and the extrapolation puts it at 91-99% of ranking error. Gemini-3.1-Pro, judging summaries, plateaus about 9 points under pass@8 while variance is still falling.
On AIME 2024, DeepConf's confidence scores already reach a similar peak. The +10 points show up on LCBv6. ThinkPRM-1.5B was trained on math, and these coding tasks sit outside that data. With the start pool fixed at 128, raising partial scores from one pass to three adds 2 to 6 points across three datasets and two models. Search still finishes short of pass@N.
RSA improves as the round count goes from 1 to 10. Growing each round from 8 to 128 candidates still cuts final accuracy by 6 points, at about 16 times the compute. On problem 120 the harness wants one complete Python program and does not run tool calls. At N=128 those calls spread from 5.8% of the initial pool to 40% of the final pool. Keeping Scorpio's top half drives them near zero and lifts that problem from 40% to 87%. The abstract's 11 points are the rollup for coding tasks.
A pipeline that already samples several answers can swap in the scorer. No separate reward model is trained. On math and puzzles the peak rises by as much as 7 points, and the stronger baseline's peak is available at 17-41% lower cost. Code verification mostly saves money on the way to a similar peak.
Further passes of the same judge buy a steadier order. Once bias flattens, those passes leave the consistently misranked pairs where they are. Search and aggregation feed the ranking into later generation, so a bad score changes more than a final one-shot pick. On problems the base model already solves, as on AIME 2024, token-probability confidence is enough.
The plateau remains under Scorpio. Across six verification settings, selection sits 14.3 points below pass@128 on average, and search keeps a gap to pass@N. An appendix experiment replaces Qwen3.5-4B's self-judgments with other models and can lower ranking bias. The main text gives no magnitude.
The bulk of the evidence is a 4B model and a 20B model. The frontier check is Gemini-3.1-Pro on Terminal-Bench-2.1, eight summaries per task. Residual bias shows up there too. That setup is thinner than scoring 128 full solutions.
Gains of up to 7 points are a math and puzzle result. Code-verification peaks sit together. The 10-point search margin uses different pool sizes: Scorpio at 8, 32, and 128, DeepConf at 16, 64, and 256, with ThinkPRM a math-trained process reward model on the other side. The 11-point aggregation figure is the abstract's rollup. The move from 40% to 87% is problem 120, under a harness that rejects tool calls.
Verification spending stops at the price of three pointwise rounds. The 91-99% share extrapolates the variance curve; the budget was not scaled another order of magnitude. Ties count as misses, because the indicator fires when the wrong score is greater than or equal to the right one. Dollars also mix two token prices from 13 September 2026, so a percent off the invoice is not a percent off the tokens.