Letter-counting tests expose Fable 5.1 false safety triggers and poor token efficiency
Developer maksymandr published a series of hands-on tests on "letter counting" tasks, showing that Fable 5.1 failed across the board on simple tasks far outside the distribution of mainstream benchmarks like SWE-Bench and TerminalBench, exposing frontier LLMs' out-of-distribution generalization failures.
Confirmed
- Completely harmless requests, such as asking Fable 5.1 to count letters in random word groups (e.g., "circulation roulette mixed"), triggered biosafety guardrails and got refused; the author compiled the rejected prompts and found the model was also downgraded to Opus as a result — calling the model "basically unusable" because of this.
- At low reasoning intensity, GPT-6-Astra maintained 93% accuracy on 1280-character texts, while Fable managed only 53%.
- Fable only engaged thinking tokens once input reached roughly 320 characters; even 20-character short strings were frequently miscounted, where not thinking could have improved accuracy — revealing flaws in its adaptive thinking mechanism.
- On 1280-character long-text tasks, Fable consumed over 3x the tokens yet scored worse; Astra achieved higher accuracy with fewer tokens.
Why it matters
- The experiment shows that frontier models that ace mainstream benchmarks can still fail badly on simple out-of-distribution tasks — a gap between benchmark scores and real-world usability.
- Guardrails mistakenly blocking harmless inputs, plus thinking-trigger thresholds and token efficiency issues, directly affect a model's practical usability and cost.
2026-09-18 ~ 2026-09-18 · 5 related posts
Primary sources
- Counting letters triggers Fable's bio safeguards, exposing frontier LLM generalization failures — maksym_andr ·
- Frontier models fail at counting letters: Astra hits 93% accuracy, Fable only 53% — maksym_andr ·
- Fable only starts thinking at 320 chars, still miscounts short passages in letter test — maksym_andr ·
- [source] Frontier models fail at counting letters: Astra hits 93% accuracy, Fable only 53% — maksym_andr · 2026-09-18
- [source] Fable only starts thinking at 320 chars, still miscounts short passages in letter test — maksym_andr · 2026-09-18
- Fable uses 3x more tokens than Astra on letter-counting yet scores worse — maksym_andr · 2026-09-18
- Fable 5.1 bio-safeguards trigger on harmless letter-counting, making it 'unusable' — maksym_andr · 2026-09-18
- [source] Counting letters triggers Fable's bio safeguards, exposing frontier LLM generalization failures — maksym_andr · 2026-09-18