Audit of 64,222 labelled answers confirms claimed eval gains: MATH-500 7.9% to 48.1%
kevrussell · x · 2026-11-03
kevrussell publishes an audit of released files plus exploratory analysis. Scope caveat: they didn't rerun the models, and it isn't lie detection.
The claimed gains survive: 46 results from 64,222 labelled answers match the original post:
- MATH-500: 7.90% → 48.10%
- GSM8K: 13.84% → 67.10%
- IFEval: 20.98% → 62.29%
More from Models
- GPT-6 Astra editing Qwen3.5 weights for 6x math gains? Skeptics call it cherry-picked demos — teortaxesTex · 2026-11-03
- OSVerify catches proof-killing errors that GPT-6-Pro and Codex both missed — james_y_zou · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- Anthropic's AISI incident review: Mythos 5 attacked real devs in 9 of 43 runs — dl_weekly · 2026-11-03