AdvancedMathBench: A Higher-Level Math Proof Benchmark
internlm · hf · 2026-07-14
The authors propose AdvancedMathBench to evaluate large models on advanced mathematical proof generation and verification. Existing math benchmarks mostly focus on high school or competition problems and often only check the final answer, making it hard to measure the actual validity of the proof process. This benchmark attempts to fill the gaps in "advanced mathematical reasoning" and "proof verification".
Benchmark Composition
- ProverBench: 296 problems covering undergraduate to PhD qualifying exam levels.
- VerifierBench: 888 model-generated proof trajectories paired with human expert ground truth, used to evaluate if models can judge a proof's validity and provide reasonable explanations.
- It also builds an automated verification pipeline trained on large-scale expert annotations, outputting correctness judgments and fine-grained error analysis, which shows strong alignment with human experts on a hold-out set.
Results
Experiments show this benchmark is still very challenging for frontier models:
- The best model scores only 75.8 and 66.1 on the UGD and QE splits for proof generation, respectively.
- The best Balanced F1 for proof verification is only 65.1.
- The overall true negative rate is low, indicating that "identifying incorrect proofs" remains a clear bottleneck.
More from Research
- Project APE finds verifier reliability drops when papers contain multiple errors — soumitrashukla9 · 2026-07-22
- Project APE says verifier costs fell about 90x in a year as Chinese open models lead — soumitrashukla9 · 2026-07-22
- OpenAI-linked paper says capability RL can make models more reward-seeking — MariusHobbhahn · 2026-07-22
- Project APE builds its verifier benchmark from 100 AI-written papers with injected errors — soumitrashukla9 · 2026-07-22
- Paper proposes a CRED taxonomy and benchmark to measure research-error detectors — soumitrashukla9 · 2026-07-22
- OpenAI says long-horizon models need safety and alignment checks across full action sequences — rhiever · 2026-07-22