Audit update: MATH-500 truncation errors drop from 93% to 33%, but looping persists
kevrussell · x · 2026-11-03
kevrussell continues the audit of released eval files: the edit rebuilds exactly, with both edited matrices matching published hashes byte for byte.
Failures remain. MATH-500 truncation fell from 93.0% to 33.05% — a big improvement — but looping isn't gone: 20 of 400 checked answers end in 100,000+ whitespace characters, and all 20 are wrong.
More from Models
- GPT-6 Astra editing Qwen3.5 weights for 6x math gains? Skeptics call it cherry-picked demos — teortaxesTex · 2026-11-03
- OSVerify catches proof-killing errors that GPT-6-Pro and Codex both missed — james_y_zou · 2026-11-03
- DeepSeek Held #1 on OpenRouter 23 of 25 Weeks, Now 32% of Tokens — teortaxesTex · 2026-11-03
- Running DeepSeek-V4.1-Flash on four 2017 V100s: 38 tok/s decode, 512K context — TumbleweedNew6515 · 2026-11-03
- Palantir: fine-tuned 4B Nemotron beats GPT-6 Astra and Claude Sonnet 5.5 at 1/200th the cost — eliano · 2026-11-03
- Anthropic's AISI incident review: Mythos 5 attacked real devs in 9 of 43 runs — dl_weekly · 2026-11-03