New MathAdv Benchmark Shows Theorem Provers Ace Problems but Fail Equivalent Reformulations
furongh · x · 2026-09-13
A UMD team led by Furong Huang introduces MathAdv (arXiv:2608.25449), a diagnostic benchmark spanning 13 math domains across four dimensions: Know, Reason, Formalize (Lean 4), and Generalize. Key findings: Goedel-Prover-V2 solved six original problems but failed all equivalent reformulations (reverse never happened); formalization remains the main bottleneck; performance varies sharply across domains; natural-language hints help general LLMs but hurt proof-specialized models. The authors stress the benchmark measures model behavior, not advancement of human mathematical understanding. Dataset and scripts are open-sourced.
Related event: MathAdv Benchmark Shows Theorem Provers Fail on Equivalent Rewrites(2 posts)→
More from Models
- Users report GPT quality drop; an old-school prompt warm-up trick to regain compliance — NickPassig · 2026-10-31
- Gemini 4 Carbon vs Opus 5.5 hands-on: 3x cheaper at $4.10 and twice as fast — vitaliychiley · 2026-10-31
- 4B model Spark-X2.6 scores 94.6/100 on community benchmark, beating 27B-35B models — mesolimbic-bliss · 2026-10-31
- opencode: $100 of usage gets you DeepSeek V4.1 Flash — intellectronica · 2026-10-31
- Researcher Finds 600+ Bugs in LLM Kernel Code; Claude Blocks Safety Analysis — pmddomingos · 2026-10-31
- Signal65: Snowflake's CoCo agent finishes workflow 4.6x faster than manual — ryanshrout · 2026-10-31