Model eval audit continues: claimed gains hold but MATH-500 anomaly persists

An audit of a model's published eval files (64,222 labeled answers) confirms the claimed improvements and byte-for-byte patch reproduction, but issues remain: MATH-500 truncation errors dropped from 93% to 33%, with 20 of 400 cases still looping.

2026-11-03 ~ 2026-11-03 · 3 related posts

• Audit of 64,222 labelled answers confirms claimed eval gains: MATH-500 7.9% to 48.1% — kevrussell · 2026-11-03
• Audit update: MATH-500 truncation errors drop from 93% to 33%, but looping persists — kevrussell · 2026-11-03
• Reproduced GPT patch passes sycophancy test 3/3, but MATH-500 still drops 93% to 33% — kevrussell · 2026-11-03