Math Verify Score bug fixed in LM Evaluation Harness; pre-2026 paper scores may need reruns
RylanSchaeffer · x · 2026-10-15
Part 8/8 of Schaeffer's thread: the team found and fixed how Math Verify Score was computed in the LM Evaluation Harness.
The author warns that any paper reporting scores before January 2026 should probably have their numbers rerun — a notable caveat for the open-source eval ecosystem.
Related event: NeurIPS Rejects Study Showing Single Contamination Breaks Evaluation Floor(4 posts)→
More from Research
- AlphaNIST: one RL policy solves all 20 NIST assembly tasks with 98.2% success in simulation — hbouammar · 2026-10-15
- Linear Depth Memory: matrix-valued memory tops PreLN and mHC on DepthBench across scales — SonglinYang4 · 2026-10-15
- Neural field planning for hybrid manufacturing cuts estimated build time 68.3% — jiqizhixin · 2026-10-15
- Tev-Gym: 1,170 dense-reward RL environments push long-horizon agentic training — RulinShao · 2026-10-15
- Claude systematically debunks the speculative EDAR-selenium hypothesis about East Asians — teortaxesTex · 2026-10-15
- Microsoft's AutoEnvScaling lets agents build their own RL environments, lifting Qwen3 Terminal-Bench from 41.1 to 53.3 — omarsar0 · 2026-10-15