Whatever Happened to Test-Time Scaling? Problems Needing pass@8192 Now Solve One-Shot
marc_lelarge · x · 2026-10-15
Marc Largouët asks what happened to test-time scaling: a year ago the field used absurdly large pass@k sampling on problems that models now one-shot. His attached table shows whole-proof generation results like DeepSeek-Prover-V2 (CoT, 671B) reaching 88.9% at pass@8192 and Kimina-Prover-Preview (72B) at 80.74%. He floats several possibilities: is test-time scaling still used in practice, was it mainly a way to mine better training data for new models, or was it never really used at all — just burning tokens?
More from Models
- SWE-Bench Multimodal Hits Vals: Every Claude Model Outperforms OpenAI's — dejavucoder · 2026-10-15
- Scale CEO spotlights Muse Wonder: untrained for 3D yet beats Gemini 4 in physics visualization tests — alexandr_wang · 2026-10-15
- Cohere tops WMT26 translation benchmark: small agentic model scores 84.36, beating Mistral and DeepL — cohere · 2026-10-15
- Anthropic has never launched a Sonnet/Opus-tier model on a Wednesday — ChrisGPT · 2026-10-15
- 8k context beats 480k: OpenAI Decisions API cuts costs 84% at 98.25% F1 — OpenAIDevs · 2026-10-15
- Gemini 4 Argon hit with HTTP 524 timeouts outside Antigravity; Google payment also glitching — cedric_chee · 2026-10-15