Claim-Level Reliability Assessment for Efficient Test-Time Reasoning
Sen Xu, Wei Wang, Shixi Liu, Jixin Min, Yingwei Dai, Zhibin Yin, Yirong Chen, Junlin Zhang
cs.AI, cs.CL
2026-08-12
CLR moves half of test-time compute from extra sampling to refuting key claims, beating self-consistency by up to 12 points while cutting tokens up to 40% on GPT-OSS.
Self-consistency is the workhorse of test-time scaling: sample N solutions to one problem and vote on the final answer. The vote counts heads, not quality, so five consistently wrong traces can outvote three correct ones. Fixes usually lean on statistical signals such as token probabilities, or on separately trained verifiers, both with limited reach.
This COLM 2026 workshop paper from Weibo takes another route: move half of the test-time budget from sampling more solutions to targeted falsification of key claims. No training, no extra model; the same model writes solutions and then audits them.
CLR runs two stages. Stage 1 samples K reasoning traces, each ending with exactly M (default 5) decision-critical claims: intermediate propositions whose failure sinks the conclusion, such as a factorization identity or a primality judgment. Stage 2 hands the original problem and the claim list, without the trace or the answer, back to the same model and asks it to attack each claim: contradictions with the problem, conflicts between claims, hidden assumptions, simple counterexamples, arithmetic errors. Each claim gets VALID or REFUTED.
Scoring is nonlinear: trace weight equals the surviving fraction raised to the M-th power. Four of five claims surviving yields 0.8 to the fifth, about 0.33; three of five drops to 0.078. The exponent makes a refuted critical claim more consequential than linear averaging would. Predictions are then grouped into equivalence classes and weighted by total score, so a reliable minority can overturn an incorrect majority.
The design rests on an asymmetry: constructing a correct solution demands a flawless chain, while refuting a wrong claim needs only one decisive flaw. Verification is cheaper than generation, which is what makes the reallocation pay off. The authors call this an inductive bias, not a guarantee.
Budget accounting: CLR@K uses K generation calls plus K verification calls, matching Cons@2K; the primary comparison is CLR@32 against Cons@64.
Four models, four math benchmarks (HMMT25/26, CMIMC25, Apex-shortlist), matched 64-call budget:
| Model (benchmark) | Cons@64 → CLR@32 accuracy | token change |
| Gemma-4-12B-it (HMMT25) | 76.67% → 88.75% | +33.4% |
| GPT-OSS-20B (CMIMC25) | 77.50% → 82.19% | −37.0% |
| GPT-OSS-120B (CMIMC25) | 75.00% → 80.00% | −21.6% |
| Qwen3.5-27B (CMIMC25) | 95.00% → 97.50% | −2.5% |
The benefit profile differs by model. Gemma-4-12B improves on all four benchmarks by 7.12 to 12.08 points at the cost of 22% to 48% more tokens. Both GPT-OSS models gain accuracy while saving 21.6% to 39.8% of tokens, though GPT-OSS-20B dips 0.42 points on HMMT25. Qwen3.5-27B starts above 90% and gains at most 2.60 points.
The ablation pins down where gains come from: appending claims after a solution actually hurts, lowering single-rollout pass@1 by 0.65 to 4.56 points. The payoff arrives in stage 2, where falsification-based reweighting adds 4.48 to 7.01 points over the same unweighted candidates.
Rescue rate quantifies overturns: among cases where self-consistency votes wrong while a correct candidate exists in the same samples, CLR fixes about 37%, spanning 16% to 48% across 16 settings. Raising M from 1 to 3 adds 3.13 to 3.79 points; M=5 keeps helping on three benchmarks while HMMT26 peaks at M=3.
For inference serving this is a drop-in aggregation layer with no weight updates: the same call budget buys either accuracy or tokens. Verification requests contain only the problem and a handful of claims, and the replies are short, which is where the GPT-OSS token savings come from. Across growing budgets, self-consistency saturates and fluctuates while CLR scales more steadily.
The broader idea, using the generator as its own falsifier, is not tied to voting; Best-of-N rescoring and search pruning could adopt the same move.
CLR only reweights already-sampled candidates; if the correct answer is absent from the K samples, nothing recovers it. All experiments sit on math benchmarks with a math-specific claim-extraction prompt, so transfer to code or factual reasoning is unverified. The authors concede the nonlinear score is a heuristic that does not assume claim independence, and M jointly varies semantic coverage and exponent sharpness, so the ablation cannot separate them. Curves cross at intermediate budgets, so CLR does not dominate everywhere. On Gemma the method costs a third more tokens, so the efficiency story is not universal.