reuF2F benchmark: 244 REU math problems formalized in Lean, best model solves only 109
wellecks · x · 2026-10-20
Researchers introduce reuF2F, addressing the verification bottleneck as LLMs scale up conjecture-proving: 244 open math problems from REU programs, expert-formalized in Lean, plus Leanimum-agent, a minimal coding agent so all models compete in the same end-to-end harness. The best model resolves only 109 of 244. A cost-vs-pass@1 chart shows Gemini 4 Argon and Claude Opus 5.5 near 40%+ at $5–15/problem, GPT-6.1 Sol hitting 40% at $1.5, while Kimi K3, Qwen3.8-Max, and Gemini 3.8 Flash stay under 10%.
Related event: reuF2F Benchmark: Best Model Solves Only 109 of 244 REU Problems(2 posts)→
More from Models
- Terminal Bench 4 audit: 44 of 65 tasks cheatable, Sonnet 5.5 hits 89.2% after fixes — scaling01 · 2026-10-20
- Skyfall debuts Sky-001, claiming 100x speed and 150x cheaper than Astra — Distinct-Question-16 · 2026-10-20
- DeepSeek-V4 answers flip on a 4-token cycle, traced to chunked KV-cache compression — Francis_YAO_ · 2026-10-20
- vLLM ecosystem ships d3-turbo: multimodal decision model, 15x faster, tops Jev Decision Index — vllm_project · 2026-10-20
- MOSS 5B architecture diagram built from official code, MIT-licensed — TheLatentExplorer · 2026-10-20
- Devs say no model loyalty: will switch to OpenAI for better limits — Angaisb_ · 2026-10-20