Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling
Mobina Kashaniyan, Ali Jannesari
cs.LG, cs.DC, cs.PF
2026-09-17
Fixing N=8 with the same vote, eight serial A100 calls use 4.64–4.86× GPU energy and 5.77–6.12× P95 latency versus one batched eight-candidate call.
Self-consistency and best-of-N treat the inference budget as a single integer N: how many candidates per prompt. N counts outputs. It does not say how those outputs were produced. Eight candidates can come from one batched generation call, or from eight sequential single-candidate calls. The vote is the same. The GPU work is not.
Kashaniyan and Jannesari at Iowa State isolate that execution choice. An audit of Self-Consistency, Adaptive-Consistency, Universal Self-Consistency, Large Language Monkeys, Scaling Test-Time Compute, and Difficulty-Adaptive Self-Consistency finds that N is almost always reported, while calls per query, candidates per call, and energy almost never are. Hardware is often missing too. Two papers can share a model, a dataset, a decoding setup, and N=8, and still have incomparable systems cost if their generation schedules differ.
On HPC clusters, LLM inference often runs as a finite batch job rather than a long-lived server. Logging, fixed seeds, and intermediate analysis also split one sampling budget across several calls. Papers usually treat that split as an implementation detail. Measured GPU energy can still move by about 4.6×.
The schedule is S=(b1,...,bC), with the bi summing to N. At N=8 they compare four splits:
Calls run one after another on the same GPU. Candidates inside a call decode as a batch. Models are Phi-3-mini-4k-instruct and Qwen2.5-1.5B-Instruct, temperature 1.0, top-p 0.95. Answers are extracted and plurality-voted; there is no verifier. GSM8K keeps a normalized numeric answer; SciQ keeps A–D. Ties take the first candidate, so N=2 falls back to N=1 when the first two disagree.
Accuracy uses 500 GSM8K prompts and prefixes of a shared eight-candidate pool for N in {1,2,3,4,8}. That pool is for quality only; its latency and energy are not assigned to each N. Systems measurements use a separate 100-prompt job with three repetitions. All four schedules sit in the same A100 job: two repetitions run 1×8 toward 8×1, one runs the reverse, to cut thermal and order effects. Energy is the NVML cumulative-counter difference over prompt processing, prefill, decoding, every generation call, extraction, and voting. Model load, warm-up, and final grading sit outside the window. The number is gross GPU-device energy: idle is not subtracted, node power is not included. Main jobs run on A100-SXM4 80GB cards, 500 W cap, graphics clock locked at 1275 MHz, through Hugging Face batch generation, not vLLM-style continuous batching. The SciQ short-output check moves to V100.
On 500 GSM8K items, raising N from 1 to 8 lifts Phi-3 from 81.4% to 89.8% (+8.4 pp, 95% paired bootstrap [5.8, 11.2]) and Qwen from 51.4% to 69.8% (+18.4 pp, [14.8, 22.0]). N=1 and N=2 match because a two-way tie keeps the first candidate. Random tie-breaking moves expected accuracy by at most 1.4 pp. Conditioning on successful extraction does not explain the gains either.
Growing the batched budget raises energy per query and cuts energy per token:
| Setup | N=1 J/query | N=8 J/query | N=1 J/token | N=8 J/token |
| Phi-3/V100 | 631 | 1286 | 2.634 | 0.655 |
| Qwen/A100 | 596 | 975 | 2.222 | 0.459 |
Mean latency goes 5.06 to 8.91 s for Phi-3 and 5.26 to 7.64 s for Qwen. GPU-hours per 1,000 queries go 1.41 to 2.47 and 1.46 to 2.12. Batching amortizes tokens. More candidates still make the query more expensive.
The schedule sweep holds N=8. Logical generated tokens differ by 0.8% (Phi-3) and 1.0% (Qwen) across schedules. Relative to 1×8:
| Schedule | Energy Phi-3 | Energy Qwen | P95 Phi-3 | P95 Qwen |
| 1×8 | 1.00 | 1.00 | 1.00 | 1.00 |
| 2×4 | 1.63 | 1.66 | 1.81 | 1.97 |
| 4×2 | 2.71 | 2.86 | 3.21 | 3.57 |
| 8×1 | 4.64 | 4.86 | 5.77 | 6.12 |
The middle two schedules climb with the number of calls. The penalty is gradual, not a cliff at full serial. Eight serial calls versus one batch of eight: 4.64× energy on Phi-3 (CI [4.48, 4.79]) and 4.86× on Qwen ([4.71, 5.02]); P95 latency 5.77× and 6.12×; throughput 16.7% and 17.9% of the batched baseline. GPU-hours per 1,000 queries rise 2.09 to 12.49 (Phi-3) and 2.13 to 11.76 (Qwen). Phi-3 mean power falls from 177.8 W to 139.8 W, but mean latency grows 5.97×, so total energy still climbs. Lower average watts is not the same as a cheaper query.
Across three independently scheduled A100 jobs, energy ratios sit in 4.43–4.64 (Phi-3) and 4.85–4.88 (Qwen); throughput stays 17–18%. Length quartiles still show energy ratios of 4.40–4.77 (Phi-3) and 4.51–5.09 (Qwen). On SciQ/V100, mean generations are 1.71 tokens for Qwen and 3.46 for Phi-3, and 8×1 still costs 2.57× (Phi-3) and 3.34× (Qwen) the energy of 1×8, with throughput at 36% and 23%. SciQ changes both GPU and output length, so the smaller ratios cannot be blamed on short outputs alone. The supported claim is narrower: the schedule tax shows up on both the long GSM8K/A100 load and the short SciQ/V100 load.
Linearly scaling the measured per-query gap to one million queries adds about 1.37 MWh and 10,401 GPU-hours for Phi-3, and 1.00 MWh and 9,631 GPU-hours for Qwen. That is an illustration of these configs, not a forecast for other stacks.
If a self-consistency loop samples one candidate per call and concatenates them, the write-up reports N=8 while the GPU pays something close to 8×1. When candidates are independent and memory fits, put them in one batch. If memory does not fit, use the largest feasible batch and the fewest calls: b=min(N, bmax), C=ceil(N/b). Tree search and interactive agents that depend on prior outputs have to stay sequential; this rule does not cover them.
For anyone writing a multi-candidate evaluation, N and accuracy are not a complete systems description. Report calls per query, candidates per call, batching mode, latency, throughput, GPU-hours, and the energy measurement boundary. Otherwise two "N=8" numbers cannot be compared.
This is a measurement warning, not a new sampler. The operational rule is usable today, on the small models and Hugging Face batching that were actually measured.
Two small models, A100 and V100, Hugging Face batch generation. The ratios should not be copied onto larger models, newer GPUs, vLLM-style continuous batching, or multi-GPU serving. In continuous serving, a request can share a batch with other traffic, so the lab assumption of one exclusive GPU for these eight candidates loosens. Candidates are independently sampled; search that depends on a previous output has to stay serial. Energy is gross GPU-device energy over the query window, with no node-level power and no idle subtraction. The end-to-end gap is measured; repeated prefill, framework call overhead, synchronization, and utilization are not isolated. SciQ changes both GPU and output length, so the smaller ratios cannot be blamed on short outputs alone.