Blind test of 12 models finds Fable 5.1 reads least like AI at 14%, Gemini 3.8 Flash worst at 77%
PawelHuryn · x · 2026-09-07
- Pawel Huryn ran 3 writing tasks × 3 tries per model with no system prompt — 108 texts, 1,647 blind pairwise judgments by four AI judges (Gemini, Grok, Opus, GPT-5.6 Sol), discarding 30% that flipped when order was swapped.
- Results (lower = less AI-sounding): Fable 5.1 14%, Grok 4.6 22%, Opus 5 24%, GPT-6 Astra 30%; GLM-5.3 55%, Kimi K3 56%, Qwen 3.8 Flash 62%; GPT-5.6 Terra 75%, Gemini 3.8 Flash 77%.
- Awkward twist: the benchmark was built with Fable 5.1, which won anyway — scored only by Google, xAI and OpenAI models since judges never score themselves.
- Human check: model LinkedIn posts beat his pre-ChatGPT (2021-22) posts 355-0. Data and texts published on GitHub within an hour.
Related event: Blind Test of 12 Models: Gemini 3.8 Most 'AI-Flavored'(3 posts)→
More from Models
- Local benchmark: Flint matches Qwen3.6 quality at 50/52 while running 3.6x faster on M4 Pro — RichardSocher · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- One stray 'u' and Claude starts spelling like a Brit — cantrell · 2026-10-27
- Cohere's Invent-A-Dataset paper: zero-seed synthetic data beats frontier APIs on quality and diversity — idanbeck · 2026-10-27
- ARC Prize redesigns leaderboard: highest reasoning effort by default, CSV export — mhmazur · 2026-10-27
- Cohere: open-weight Qwen3.8-Flash-Next tried uploading probe file to Beijing cloud bucket in sandbox — wstone_bd · 2026-10-27