8 serial calls burn 4.86x GPU energy vs one batched call in test-time scaling
iowastate · hf · 2026-09-19
This paper shows candidate count N alone doesn't describe the systems cost of test-time scaling. On 500 GSM8K prompts, raising N from 1 to 8 lifted accuracy by 8.4pp (Phi-3-mini) and 18.4pp (Qwen2.5-1.5B). But fixing N=8 and comparing schedules (1x8, 2x4, 4x2, 8x1), eight serial A100 calls used 4.64–4.86x the gross GPU energy and had 5.77–6.12x the P95 latency of one batched call — consistent across three nodes and a SciQ/V100 setup. When candidates are independent and memory allows, fewer calls with larger batches are far more efficient; evaluations should report schedule and GPU-level metrics alongside accuracy.
More from Infra
- Cerebras launches Money Agent, a finance assistant powered by Qwen3 27B — irinarish · 2026-09-19
- Cloudflare saved another 100TB of RAM by reworking consistent hashing in Rust — Cloudflare Blog · 2026-09-19
- Magnitude: free open-source desktop engine profiles your hardware, picks and tunes local models — nickbaumann_ · 2026-09-19
- ContinuityBench: stateful LLM failover keeps 99.2% context vs near-0% stateless — its_vayishu · 2026-09-19
- AI Data Center Power Bottleneck Makes Bloom Energy This Analyst's Top 2026 Stock Pick — Beth_Kindig · 2026-09-19
- Jev classifies the entire Bluesky firehose for animal posts in near real time — danshipper · 2026-09-19