AA-Omniscience benchmark: 6,000 fact questions show top models still err ~25% without search
randal_olson · x · 2026-09-30
Randall Olson shared Artificial Analysis' AA-Omniscience data: 6,000 questions about specific facts across 42 work topics, answered without search. Wrong answers are measured as a share of answers given (declined questions excluded).
Headline finding: even the best models get roughly 1 in 4 answers wrong without search — if a fact matters, have the model look it up.
He also released evident-charts, an open source agent skill for Claude Code, Codex, Cursor and more that teaches AI coding agents to make clear, honest charts: it checks data first (totals mixed with parts, duplicate rows, placeholder codes, preliminary months) and critiques and rebuilds bad drafts — e.g., raw counts on a log scale, the EU total ranked among its own members, meaningless rainbow palettes.
Related event: Top AI Models Get About a Quarter of Factual Questions Wrong Without Search(2 posts)→
More from Models
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- Nathan Lambert calls out Ethan Mollick for 'selectively ignoring' Kimi in model comparisons — natolambert · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- Unsloth Q8 crushes default Q3 in a self-hosted test over 331 financial PDFs — No_Cartographer3953 · 2026-11-16