reuF2F benchmark: 244 Lean-formalized open math problems, best LLM solves only 109
wellecks · x · 2026-10-23
A new benchmark, reuF2F, collects 244 open math problems from Research Experiences for Undergraduates (REU) programs across a broad range of math areas, all formalized in Lean by experts, to test AI agents end-to-end on open problems. The team also built Leanimum-agent, a minimal coding agent for Lean, so all models compete under the same harness. Even the best-performing model resolves only 109 of 244 problems—underscoring that verification is becoming the bottleneck as LLMs scale up conjecture proving.
More from Research
- 20+ ICLR 2027 Submissions Found With Fabricated Results, Sparking Reproducibility Critique — srchvrs · 2026-10-23
- Qdrant's 64-byte-per-page recipe: 128x compression with 99.2% retrieval quality — qdrant_engine · 2026-10-23
- Qdrant benchmarks Cohere Embed 5 Pro: one dense vector per PDF page vs ColPali — qdrant_engine · 2026-10-23
- 20+ ICLR 2027 submissions self-confess fabricated placeholder results — thegautamkamath · 2026-10-23
- Pure FP vs C: Roc's DEFLATE grind closes in on libdeflate's performance — johnlindquist · 2026-10-23
- rosinality Fixes Numerical Precision Issues in Softmax, Praised by Lucas Beyer — giffmana · 2026-10-23