Private benchmark: no major LLM scores above 80% on following instructions
crystalkalem · reddit · 2026-08-27
A Reddit user ran a private benchmark testing whether models can retain and follow all 100 rules across a multi-chapter writing task, with 90% as the pass line. Every model failed: Gemini ranged 54%-72%, Claude versions mostly 65%-80%, GPT versions 58%-78%.
The author notes formatting didn't matter—precise pseudo-code or a run-on block of text produced the same failures, with over 20% of instructions dropped. A human could score 100% effortlessly. He questions trusting AI with any coding work when it forgets a fifth of given rules, and won't release the benchmark to avoid training-data contamination.
More from Models
- Meta open-sources Muse Spark Open: 816B-parameter model that runs locally on two Mac Studios — gabriberton · 2026-10-16
- Grok Voice 3.0 Tops Design Arena Speech Leaderboard, Up 7.4 Points — XFreeze · 2026-10-16
- Robotics LLMs tested on 16 tasks: GPT-6 Astra hits 17.2% success vs Fable's 9.1% — pabbeel · 2026-10-16
- Meta open-sources Muse Spark: top US open model, scores just below DeepSeek — bindureddy · 2026-10-16
- TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
- Meta's Muse Spark Open ships detailed eval docs; researcher pushes machine-readable EEE standard — evijit · 2026-10-16