Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B
Iliya Mirzaei
cs.CL, cs.AI, cs.LG
2026-07-31
Given the same token budget, none of seven test-time methods beats plain repeated sampling; self-critique and self-verification reliably lose accuracy below 7B parameters.
Methods that make a model "think harder" (Self-Refine, Reflexion, Best-of-N, multi-agent debate) almost all make it write more text. A single chain of thought might be 300 tokens; three rounds of self-critique run to 1,600; a two-round debate among three copies reaches 2,500. The compute budget has changed five- or tenfold, and spending more compute on plain repeated sampling (self-consistency) already buys accuracy on its own. So beating a single chain of thought proves nothing about whether the method's idea is what helped.
Wang et al. (2024) made this argument but gave only point estimates: no confidence intervals, no significance tests, 100 questions per dataset, with differences below roughly ten points indistinguishable from noise. This paper turns it into a proper designed experiment.
The setup is Qwen2.5 at 1.5B, 3B and 7B (Q80 quantization, CPU only, no GPU), 150 questions each from GSM8K and MATH-500, comparing seven methods: Chain-of-Thought, Plan-and-Solve, Self-Refine, Reflexion (plus a forced variant), Best-of-N (model picks), and multi-agent debate.
The crux is cost accounting. Every token a method generates for one question is charged, including critiques, reflections, debate turns and verification that never reach the final answer. That measured cost is looked up on the self-consistency curve (accuracy vs. cost at N=1 to 16) and the method is compared against the point of equal cost. To keep this affordable, one pool of 16 chains is drawn per question, N are subsampled without replacement for a majority vote, repeated 200 times, yielding the whole curve cheaply.
Comparisons are paired by question, reported with paired-bootstrap 95% intervals and a Holm correction for testing seven methods at once. The design reliably detects differences of about five to six points.
Across 36 method-by-setting comparisons, no method is significantly better than equal-cost repeated sampling in any setting; ten are significantly worse, all of them methods that ask the model to inspect its own output.
The cleanest cut comes from Best-of-N. It draws eight samples and asks the model to pick the best. Taking those same eight samples and counting the most common answer instead, with identical samples, tokens and model and only the final step differing, wins everywhere, and the gap shrinks with scale:
| Setting | model picks | counting | gap (pp) |
| 1.5B / GSM8K | 71.3% | 79.3% | -8.0 |
| 1.5B / MATH-500 | 46.7% | 58.0% | -11.3 |
| 3B / MATH-500 | 52.0% | 69.3% | -17.3 |
| 7B / GSM8K | 90.7% | 92.7% | -2.0 (n.s.) |
Below 7B, counting wins all four settings; at 7B the gap falls to about two points and is no longer distinguishable from zero. Rewriting methods (Self-Refine, forced Reflexion) stay 3.6 to 10.1 points below baseline even at 7B, with no recovery.
There is an ironic detail. Reflexion, implemented as published, never triggered a single retry on the 1.5B model: it judged itself correct on every question and quietly collapsed into a single chain of thought, scoring well precisely because it was cheap. The authors instrumented how often the adaptive part actually fires, or this number would have been reported as evidence that reflection works.
For practitioners the takeaway is practical: on automatically checkable math, with a fixed budget, spending extra tokens on one more sample beats spending them on self-critique, and the smaller the model the more this holds. If a pipeline under 7B uses Self-Refine or model-chosen Best-of-N to save money or gain accuracy, run equal-cost repeated sampling as the control first; it may be compute wasted. The claim has a hard boundary, though, set out below.
The direction at scale is informative. The gap closes at 7B because the verifier increasingly agrees with the majority, not because it becomes a better judge; when it departs from the majority it is still usually wrong. Trained verifiers (process supervision, trained self-verifiers) do beat majority voting at about this scale in the literature, which suggests training, not the act of verifying, is what makes a verifier worth its tokens.
The authors state the boundaries plainly, which is part of why the result is credible. The full cost-matched comparison covers only 1.5B and 3B; Best-of-N extends to 7B; frontier models are untested, so whether judging overtakes counting above 7B is unknown. The task is mathematics with checkable answers, while Self-Refine was designed for open-ended generation: on math an answer is simply right or wrong, so every rewrite is a gamble, and on open-ended tasks the majority-voting baseline does not even exist (you cannot take a majority over prose). The authors explicitly do not claim these methods do not work, only that on math at these sizes they lose to spending the same tokens on more sampling.
Each method has one implementation with fixed hyperparameters and no per-dataset tuning, and prompt wording matters (Reflexion firing zero times on 1.5B depends on the judging prompt). Statistically, only question-level variance is estimated, not seed-level, so intervals for the three random methods are lower bounds on total uncertainty and the significant results should be read as the least secure part. The authors also disclose two bugs they caught: a server-side context truncation that biased Best-of-N toward easier questions (re-run and refilled), and a scoring bug that silently scored Best-of-N as majority voting (after the fix it went from "on the baseline" to "below baseline in every setting"). An independent re-derivation script agrees on all 96 checks.