Explore Broadly, Reason Sharply: Push Small Models toward the Frontier via Sampling
Panagiotis Theodoropoulos, Nan Jiang, Xintong Duan, Ali Hasan, Yuriy Nevmyvaka, Evangelos A. Theodorou, Wei Deng
cs.LG
2026-09-30
PPT runs a ladder of sharpened sampling chains that swap reasoning traces, beating GRPO without training and letting a frozen Qwen3.5-9B edge past GPT-5 on GPQA and LiveCodeBench.
RL post-training with verifiable rewards is the standard recipe for reasoning models, but it needs a scorer, costly gradient updates, and optimizing a narrow set of rewarded tasks erodes prior abilities, producing what the literature calls 'jagged' generalization. For open-ended scientific inquiry or long-horizon planning there is often no reliable reward at all. A recent training-free alternative samples from a power-sharpened distribution πα ∝ p0^α, amplifying high-likelihood complete answers on the bet that good reasoning traces are already latent in the base model. Single-power sharpening hits a structural wall, though: crank α up and the sampler gets trapped in plausible-but-wrong trajectories; keep α low and the answer distribution stays diffuse. On the stronger Qwen3.5-9B, single-chain Power Sampling actually falls below standard decoding on most benchmarks.
PPT (Parallel Power Tempering) ports parallel tempering from multimodal MCMC to sequence-level LLM sampling:
The second contribution is a bug fix to prior power samplers: a one-way truncation bias. Early-stopped implementations let proposals shorten a record but never lengthen it, because regeneration is capped at the current length; the reverse move has zero probability, yet the implementation accepts it using token scores truncated at the candidate's endpoint. The authors prove this creates uncompensated probability flow toward short records with an asymptotic bias floor that parallel tempering cannot repair. Their fix is a fixed-horizon construction: pad deterministically after EOS up to a cap T, so a resample starting before the old EOS can regenerate past it. Under this construction every MH step preserves the target exactly.
Ladder design matters: adjacent rungs should have roughly equal swap acceptance (equi-accepting), since the weakest pair throttles transport along the whole ladder. In practice they initialize a geometric ladder and tune from measured acceptance rates.
Qwen3-8B, Pass@1 (%):
| Method | MATH500 | GPQA | AIME 24&25 | LCB v5 |
| Standard decoding | 83.9 | 55.8 | 72.7 | 48.1 |
| Power Sampling | 82.2 | 57.6 | 73.3 | 53.5 |
| GRPO | 86.4 | 53.0 | 74.0 | 54.1 |
| PPT | 88.0 | 60.1 | 78.3 | 58.4 |
PPT is best or tied-best in all 15 model-benchmark settings, the only tie being saturated GSM8K. Note GRPO scores 53.0 on GPQA, below standard decoding's 55.8, a direct sample of RL capability erosion. On Qwen3.5-9B:
| Method | GPQA | AIME 24&25 | LCB v5 |
| Standard decoding | 82.4 | 82.3 | 81.9 |
| GPT-5 (reported) | 85.4 | 95.0 | 84.6 |
| Opus 4.5 (reported) | 86.6 | 93.3 | 87.1 |
| GLM 4.7 (reported) | 85.9 | 95.0 | 89.4 |
| PPT | 85.9 | 93.3 | 84.9 |
PPT ties GLM 4.7 and beats GPT-5 on GPQA, edges past GPT-5 on LCB v5, and trails GPT-5 by 1.7 points on AIME. A 9B open model reaching this zone purely through decoding strategy is the headline number.
Two controls pin down where the gains come from. With the single chain's MCMC budget raised to match total decoded tokens, PPT still leads by 3.5 points on LCB v5 and 3.0 on AIME; running the same K chains uncoupled, with no swaps, hurts every setting under both Pass@1 and majority voting. The gains come from the exchange mechanism, not from running more chains. Accuracy saturates around K=4-5 replicas, with K=4 gaining 2.7 and 3.9 points over a single chain; thanks to batching, K=4 costs only 2.1x (MATH500) and 1.2x (HumanEval) a single chain's wall-clock. Swap overhead is 0.037s per problem, about 0.1% of total time, while per-example local-update time goes from 0.86s to 3.21s (8B, K=4) and peak KV-cache memory on the 9B model from 43.8 to 75.4 GiB.
A training-free, reward-free inference-time boost that works exactly where RL post-training cannot: frozen models and open-ended tasks with no verifier. For the research line arguing that base models already contain strong reasoning, this is the most complete account so far, with a bias fix and matched-compute controls. The honest framing: it trades inference compute for accuracy, so latency-sensitive serving is not the use case; offline batch inference and high-value single calls are.
The authors flag that the convergence guarantee is asymptotic and worst-case (the contraction coefficient can be tiny), so it does not certify small error at practical budgets; frontier comparisons use reported scores rather than controlled re-runs; only Qwen3-family models at 4B-9B were tested. Further concerns from a close read: the power target uses base-model likelihood as a quality proxy, with no guarantee on tasks where likelihood and correctness decouple; the whole argument leans on capabilities already being latent in the base model, which sharpening cannot create; and how the GRPO baseline's training compute was matched is not detailed in the main text, which the strength of the 'beats RL' claim depends on. The multiplied inference time and memory keep it away from cheap everyday deployment.