DeepSeek ranks near-zero on AA-Omniscience hallucination test, sparking leaderboard credibility fight
teortaxesTex · x · 2026-10-31
Chinese AI commentator @xiangning0 blasted Artificial Analysis' updated AA-Omniscience benchmark as a "pay-to-play joke list" after DeepSeek V4.1 Flash scored just 1 on the hallucination test and an unreleased Google model topped the chart. teortaxes pushed back hard: DeepSeek has always scored near or below zero on this benchmark due to mediocre accuracy (46%) and very high hallucination rates. Per AA's own post, the V4.1 Flash 1017 update actually improved non-hallucination rate 29 points to 33% and lifted the index from -5 to +5 — still last among 107 models, vs. Claude Sonnet's 68% accuracy / 11% hallucination. The spat highlights ongoing disputes over hallucination benchmark methodology.
More from Models
- ChatGPT Plus users grumble as Pro gets 2x limits: 'Paid $20 to watch them have twice the fun' — Intelligent-Pass6641 · 2026-10-31
- 19 LLMs play 376 rounds of a Codenames-style game: guessing matters more than clueing — Hardicecream17 · 2026-10-31
- Two weeks of agent logs: INT4 quantization triples Qwen 27B's reasoning loops at long context — Old_Ad_6033 · 2026-10-31
- Devs Rageward-Hacked: LLMs Bias Toward New Code, Wreck Real Projects — yacineMTB · 2026-10-31
- Leaker Account Teases Major Unannounced OpenAI Drop From Sam Altman — iruletheworldmo · 2026-10-31
- Blogger argues $30K local Kimi K3.1 setup is worth it, despite taking 4.76 years to match GPT-6.1 Terra output — bookwormengr · 2026-10-31