Private benchmark: no major LLM scores above 80% on following instructions
crystalkalem · reddit · 2026-08-27
A Reddit user ran a private benchmark testing whether models can retain and follow all 100 rules across a multi-chapter writing task, with 90% as the pass line. Every model failed: Gemini ranged 54%-72%, Claude versions mostly 65%-80%, GPT versions 58%-78%.
The author notes formatting didn't matter—precise pseudo-code or a run-on block of text produced the same failures, with over 20% of instructions dropped. A human could score 100% effortlessly. He questions trusting AI with any coding work when it forgets a fifth of given rules, and won't release the benchmark to avoid training-data contamination.
More from Models
- Nat Lambert: wave of AI releases rumored — thinky, muse spark, Fable 5.5, OpenAI — natolambert · 2026-10-15
- Musk amplifies dev's verdict: Grok Bot easier to use than Codex and Claude — elonmusk · 2026-10-15
- Rumor: New Anthropic model imminent, apparently behind a batch of high-severity Chrome CVEs — matthew_d_green · 2026-10-15
- StartLux-Decision debuts RL-trained calibrated decision models, 27B scores 72.8 — startlux-models · 2026-10-15
- Meta's Muse Code slammed for harsh quotas: 21% left after 5 hours — ThomasScialom · 2026-10-15
- Claude Code error mentions 'Fable 5.1' and 'Opus 5', hinting at unreleased models — adonis_singh · 2026-10-15