AI-slop test details: human posts beaten 355-0, 30% of judgments discarded for order-flipping
PawelHuryn · x · 2026-09-07
- Pawel Huryn shared methodology details: tasks capped at 500 words; the human check pitted each model's LinkedIn output against 15 of his pre-ChatGPT (2021-22) posts — models won 355 judgments to 0, with 5 flips, an outcome the author found uncomfortable to present.
- 30% of all judgments flipped when pair order was swapped and were discarded; judges agreeing on kept picks matched 89-97% of the time.
- Caveat: xAI's API injects a hidden system prompt, so Grok was the only model not run on an empty system prompt.
Related event: Blind Test of 12 Models: Gemini 3.8 Most 'AI-Flavored'(3 posts)→
More from Models
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27
- Open Models Match GPT-6 on MedMCQA but Trail 17 Points on Hard Clinical Cases — keunwoochoi · 2026-10-27