REUF2F benchmarks frontier models on 244 open math problems; Gemini 4 Argon leads at 44.7%
wellecks · x · 2026-10-20
The new REUF2F benchmark tests frontier models on 244 undergraduate-level open research math problems that can be proved or refuted — a sweet spot where frontier agents do nontrivially well but aren't perfect at pass@2, and open-weight models lag far behind.
Key results:
- Gemini 4 Argon (high) leads at 44.67% pass@2, followed by Claude Opus 5.5 (max) at 43.85%, GPT-6.1 Sol at 40.98%, GPT-6 Astra at 37.70%; Kimi K3 manages just 9.84% and Qwen3.8-Max 9.02%.
- On cost, GPT-6.1 Sol at medium effort hits near-frontier accuracy at $1-2/problem, topping the Pareto frontier; Kimi K3 and Qwen3.8-Max cost $20+ per problem with poor accuracy.
- By subject, algebra/algebraic geometry and discrete math are strongest (25/45); 'Others' is hardest (≤7). Gemini 4 Argon is notably strong at refuting statements.
- Test-time scaling curves rise roughly log-linearly with token budget but plateau near 45%.
More from Models
- Private evals: pplx-decider shows 40% recall gain over Jev decision models — marktenenholtz · 2026-10-20
- DeepSeek V4.1 refuses jailbreak persona, spotting prompt injection in red-team test — teortaxesTex · 2026-10-20
- Dev claims ~16% extra speedup for Qwen models with MTP, open llama.cpp/mlx release planned — karminski3 · 2026-10-20
- Hazy Research's open-source NCM generates text 31x faster via coarse-to-fine concept modeling — simran_s_arora · 2026-10-20
- Blogger claims DeepSeek's unreleased model nearly matches Opus at design, shares one-prompt demo — sven_ai · 2026-10-20
- Fingerprint test flags DeepSeek ds-4.1-flash as likely GPT-6-astra — sven_ai · 2026-10-20