ICML Paper: MATH-Perturb Reveals LLMs Rely on Pattern Matching in Math Reasoning
MengdiWang10 · x · 2026-08-25
Researchers from Princeton University and Google have released the MATH-Perturb benchmark and an accompanying paper (ICML 2025) to test the true mathematical reasoning capabilities of LLMs.
The study finds that when familiar math benchmark problems undergo small but fundamental structural changes—termed "Hard Perturbations"—model performance drops significantly. LLMs often continue down familiar solution paths (modes) rather than adapting to the new structure. This suggests that LLM performance on math tasks relies heavily on pattern matching and shortcuts rather than genuine generalizable reasoning. The benchmark consists of 279 perturbed problems derived from the hardest level-5 problems in the MATH dataset.
More from Research
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- 8 non-negotiable principles for building AI that actually learns from experience — tokenbender · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15