reuF2F benchmark: frontier agents solve only ~44% of 244 formalized open REU math problems
A multi-institution team from CMU, CUHK-Shenzhen, Tsinghua, and others (led by Zeyu Zheng, per the post) released the reuF2F benchmark: 244 open mathematics problems drawn from the US REU (Research Experiences for Undergraduates) program, all formalized in Lean 4 by experts and spanning a broad range of mathematical subfields. The team also built and open-sourced the accompanying Leanimum Agent.
Confirmed
- Motivation: as LLMs begin proving mathematical conjectures at scale, verification has become the bottleneck, and Lean formalization offers a way out.
- Benchmark design: the problems are undergraduate-level open research questions where models may either prove or disprove the statement, rather than having a single standard answer; the team believes this difficulty best differentiates models at a moderate budget.
- Results: frontier models solved just over 100 problems in total (around 109); Gemini 4 Argon topped the chart with 44.7% pass@2; frontier agents performed solidly but imperfectly, while open-weight models still lag far behind.
- The team released the paper, the benchmark, and the Leanimum Agent, but withheld the solved proofs to prevent data contamination—a practice @wellecks views as good hygiene.
Why It Matters
- The benchmark directly tests genuinely open research problems rather than already-known ones, offering a more realistic measure of how far math agents can go and how large the gap with open-weight models remains, while keeping future evaluations trustworthy by not publishing the answers.
2026-10-20 ~ 2026-10-20 · 5 related posts
Primary sources
- reuF2F benchmark: 244 REU math problems formalized in Lean, best model solves only 109 — wellecks · 2026-10-20
- [source] REUF2F benchmarks frontier models on 244 open math problems; Gemini 4 Argon leads at 44.7% — wellecks · 2026-10-20
- [source] AI resolves 100+ of 244 open research problems; proofs withheld to avoid contamination — wellecks · 2026-10-20
- AI agent resolves 100+ of 244 open math problems; proofs withheld to avoid contamination — wellecks · 2026-10-20
- reuF2F: 244 open math problems formalized in Lean, best coding agent solves only ~40% — AkariAsai · 2026-10-20