Meta Releases AutoformBench: 341-Task Lean Formalization Benchmark With Cost Curves
ChrSzegedy · x · 2026-10-13
Meta's team released AutoformBench, a Lean 4 formalization benchmark pushing frontier models to their limits on graduate- and research-level mathematics.
- Motivation: Only 42% of results in openai/math could be formalized pre-release, and the cost of formalizing major theorems was unknown — key info if formalization enters peer review.
- Composition: 341 proving tasks drawn from diverse graduate/research-level sources, formalized by human experts.
- Cost curves: An accompanying chart compares tasks solved (out of 341) across per-task spend tiers ($1–$200) for multiple frontier models, exposing large cost-efficiency gaps.
A rare public benchmark pairing formalization capability with dollar cost.
More from Research
- Preparing post-training data for coding LLM fine-tunes is 'black art', says practitioner — ayushthakur0 · 2026-10-13
- JQD:DM launches Special Workflow Issue with shared skill-file LLM gatekeeping in peer review — manoelribeiro · 2026-10-13
- Depth vs hold: behavior alone can't diagnose why LLMs fail, new paper argues — burny_tech · 2026-10-13
- BudgetPix: adaptive tokenization matches full-compute pixel diffusion with 25% of the compute — _akhaliq · 2026-10-13
- SoftwareGym synthesizes GUI environments for computer-use agents, lifting OSWorld 37% relative — _akhaliq · 2026-10-13
- LightOn's open 2B model cuts Claude Code token bills by 34% via local semantic code search — lmoroney · 2026-10-13