AdvancedMathBench: A Higher-Level Math Proof Benchmark
internlm · hf · 2026-07-14
The authors propose AdvancedMathBench to evaluate large models on advanced mathematical proof generation and verification. Existing math benchmarks mostly focus on high school or competition problems and often only check the final answer, making it hard to measure the actual validity of the proof process. This benchmark attempts to fill the gaps in "advanced mathematical reasoning" and "proof verification".
Benchmark Composition
- ProverBench: 296 problems covering undergraduate to PhD qualifying exam levels.
- VerifierBench: 888 model-generated proof trajectories paired with human expert ground truth, used to evaluate if models can judge a proof's validity and provide reasonable explanations.
- It also builds an automated verification pipeline trained on large-scale expert annotations, outputting correctness judgments and fine-grained error analysis, which shows strong alignment with human experts on a hold-out set.
Results
Experiments show this benchmark is still very challenging for frontier models:
- The best model scores only 75.8 and 66.1 on the UGD and QE splits for proof generation, respectively.
- The best Balanced F1 for proof verification is only 65.1.
- The overall true negative rate is low, indicating that "identifying incorrect proofs" remains a clear bottleneck.
More from Research
- Researchers identify "Forking": train loss falls while validation loss turns up — ZimingLiu11 · 2026-10-26
- Google DeepMind pilots world's first double-blind evaluations for frontier models — scychan_brains · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- FLASH: GPU-native deformable sim learns cloth folding in minutes on one RTX 5090 — ChongZzZhang · 2026-10-26
- LEMON: zero-shot LLM misbehavior monitor flags hallucinations from attention maps — yoavartzi · 2026-10-26
- Behind the EMNLP keynote: LMs as simulated learners to generate child language acquisition hypotheses — najoungkim · 2026-10-26