reuF2F: 244 open math problems formalized in Lean, best coding agent solves only ~40%
AkariAsai · x · 2026-10-20
Researchers from CMU, CUHK-Shenzhen, Tsinghua and others introduce reuF2F, a Lean 4 benchmark of 244 formalized open problems drawn from Research Experiences for Undergraduates (REU) programs.
- Task: agents receive the Lean statement of an open problem and must produce a formal proof checked by the Lean kernel; each problem and its negation are placed in a Lean project inside a Docker container
- Why REU problems: these lesser-known open problems attract little attention, making them better suited for blind evaluation than famous conjectures
- Unified harness: the team built Leanimum-agent, a minimal coding agent based on mini-SWE-agent, so all models are compared under the same setup
- Result: even the best-performing model resolves only 40% (109/244) of problems, leaving substantial room for improvement in end-to-end formal open-problem research
The authors argue verification is becoming the bottleneck as LLMs start proving conjectures at scale, and formal mathematics offers a trusted, machine-checkable framework.
More from coding & agent
- REA gains 214K stars in a week, tops GitHub's fastest-growing AI repos — sven_ai · 2026-10-20
- Dev goes 'graph pilled': graphs will anchor every company's world model — _AustinCalvert_ · 2026-10-20
- RuneBench: Benchmarking AI Coding Agents by Playing RuneScape — charliermarsh · 2026-10-20
- A 5-step loop for debugging AI agents in production: trace everything, watch for silent failures — Tracify · 2026-10-20
- New Claude Cookbook: Auto-Review Agent Session Transcripts for Security Issues — brada · 2026-10-20
- Conversational Video Editing: Driving CapCut Web with Codex for Marketing Videos — Div_pradeep · 2026-10-20