Top models follow only ~68% of 500 instructions: measuring LLM behavior under load

Decent-Conclusion-83 · reddit · 2026-11-22

Context windows hold hundreds of thousands of tokens, but how many instructions can a model actually follow at once? Citing the IFScale study: across 20 models given up to 500 instructions, even the best followed only 68%. Other research shows system-prompt adherence decays over a conversation, and even irrelevant text reduces task accuracy.

Developers see this daily: rules in CLAUDE.md get ignored, original requirements forgotten as conversations grow, quality drops when dozens of mostly-unused tools are connected. Current workarounds — shorter rule files, clearing context, loading only needed tools — remain rules of thumb.

The author argues a fundamental gap exists: we benchmark speed, cost, reasoning, and context length, but have no accepted way to measure how instruction-following reliability degrades under load, compare degradation curves, or predict whether a prompt exceeds a model's capacity.

He proposes treating degradation as a measurable property — load in defined units, per-instruction outcomes, degradation laws, quantified limits — and introduces IEC, an open-source experimental instrument with public code and measurement data (HF blog + arXiv paper).

Original post →

More from Models

Models channel →