AA-Omniscience Eval: Sol and V4 Lead in Knowledge but Are 'Bullshit Machines'
teortaxesTex · x · 2026-07-21
A user shared interesting findings from the AA-Omniscience evaluation:
- Sol is close to Fable and categorically above K3 and V4 in knowledge.
- V4 and K3 are very close in performance.
- Grok 1.5T performs great.
- V4 is a 'bullshit machine', and so is Sol.
These results prompt a discussion on the actual effects of scaling on model hallucinations and knowledge retention.
Related event: AA-Omniscience Chart Compares Accuracy and Hallucination Rates of 28 Models(2 posts)→
More from Models
- MissionReport benchmark: only 3 of 540 AI-generated government reports pass all criteria — Exp_Mark · 2026-10-27
- Local beats cloud: Qwen 3.8 FN handles private student data as well as frontier models — dr_lm · 2026-10-27
- Aidan Gomez: model's CoT said no tool calls right before making them — aidangomez · 2026-10-27
- Nat Lambert: Cohere's model is narrow RL training on internal infra ill-suited for open weights — natolambert · 2026-10-27
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27