AA-Omniscience Eval: Sol and V4 Lead in Knowledge but Are 'Bullshit Machines'
teortaxesTex · x · 2026-07-21
A user shared interesting findings from the AA-Omniscience evaluation:
- Sol is close to Fable and categorically above K3 and V4 in knowledge.
- V4 and K3 are very close in performance.
- Grok 1.5T performs great.
- V4 is a 'bullshit machine', and so is Sol.
These results prompt a discussion on the actual effects of scaling on model hallucinations and knowledge retention.
Related event: AA-Omniscience Chart Compares Accuracy and Hallucination Rates of 28 Models(2 posts)→
More from Models
- DeepSWE Eval: Kimi K3 Matches Claude Fable 5 at 35% of the Cost — togethercompute · 2026-07-22
- Gemini 3.5 Flash Outperforms GPT-5.6 in Light Coding Tasks — Shick_hydro · 2026-07-22
- NVIDIA says Nemotron 3 Ultra scored 30/42 on the 2026 IMO problems — NVIDIAAI · 2026-07-22
- Gemma-4-26B-a4B reportedly beats Qwen3.6 and Qwen3.5 MoE fine-tunes — JLeonsarmiento · 2026-07-22
- OpenAI is reportedly briefing U.S. lawmakers on its next model family — kimmonismus · 2026-07-22
- Muse Spark 1.1 lands at 1495 on Text Arena with standout agentic-coding price performance — ycombinator · 2026-07-22