Top models follow only ~68% of 500 instructions: measuring LLM behavior under load
Decent-Conclusion-83 · reddit · 2026-11-22
Context windows hold hundreds of thousands of tokens, but how many instructions can a model actually follow at once? Citing the IFScale study: across 20 models given up to 500 instructions, even the best followed only 68%. Other research shows system-prompt adherence decays over a conversation, and even irrelevant text reduces task accuracy.
Developers see this daily: rules in CLAUDE.md get ignored, original requirements forgotten as conversations grow, quality drops when dozens of mostly-unused tools are connected. Current workarounds — shorter rule files, clearing context, loading only needed tools — remain rules of thumb.
The author argues a fundamental gap exists: we benchmark speed, cost, reasoning, and context length, but have no accepted way to measure how instruction-following reliability degrades under load, compare degradation curves, or predict whether a prompt exceeds a model's capacity.
He proposes treating degradation as a measurable property — load in defined units, per-instruction outcomes, degradation laws, quantified limits — and introduces IEC, an open-source experimental instrument with public code and measurement data (HF blog + arXiv paper).
More from Models
- Cryptographer warns frontier AI math is already good enough to break cryptography — rickasaurus · 2026-11-22
- Tesla FSD V15 Brakes Early for Pedestrians in Heavy Fog, Driver Calls It Life-Saving — yunta_tsai · 2026-11-22
- Ant-backed inclusionAI open-sources LingSante-3.1-flash, a 560B MoE medical model with 1M context — jacek2023 · 2026-11-22
- Inkling 2M compared to Mistral 7B: "we didn't know what this weight class allowed" — teortaxesTex · 2026-11-22
- Tesla FSD V15 live-perception demo in downtown Toronto sparks debate — gnukeith · 2026-11-22
- Local test: Inkling 2 mini runs 5-10x slower than Qwen and fails agent tasks — Mrinohk · 2026-11-22