Correction: newer model gets worse on the extreme-hard tail, pass@5 crossover predicted
teortaxesTex · x · 2026-10-31
A technical self-correction on a model eval analysis: the pass@1 shift (9.83%→7.11%) applies to tasks below 50% success, not below 10%; p<10% moved 2.37%→3.16% and p<0.1% moved 0.071%→0.47%. More importantly, the author flipped the sign on the tail direction: per their beta distributions, the 0910 model would 'never solve' (p<0.001) 1 of 1507 problems versus 7 for 1017 — meaning the newer model is worse on the extreme-hard tail, possibly due to entropy loss or distillation. This implies 0911 has higher pass@n with a crossover around pass@5, which the author notes is checkable.
More from Models
- SenseNova-U1.5-8B hits 125 tok/s on M5 Ultra in llama.cpp and spots a $1 receipt error — Binary_orchid · 2026-10-31
- Pundit: Anthropic killing 5-hour limit plus Fable launch makes it unstoppable — iruletheworldmo · 2026-10-31
- ChatGPT Daybreak Blue sessions may train OpenAI models even with training opt-out off — littmath · 2026-10-31
- Claude refuses to draw its own logo over copyright concerns — 141_1337 · 2026-10-31
- AI cracks a 27-year-old pillar of computer science, OpenAI and Anthropic claim — Fcking_Chuck · 2026-10-31
- Perplexity early evals: Gemini 4 Carbon tops WANDR benchmark for long-form research — AravSrinivas · 2026-10-31