Consistency separates frontier models: Kimi K3.1 Pro passes 45% of VIALS tasks 3/3 vs Astra's 63%
ArtificialAnlys · x · 2026-11-07
Artificial Analysis notes that solving a task once doesn't mean consistent performance — consistency is a key frontier-model advantage on VIALS-AA. Across 161 life-sciences visual reasoning tasks attempted three times each, Kimi K3.1 Pro answered 119 of 161 correctly at least once (GPT-6 Astra: 118), but requiring all three attempts to pass flips the picture: Kimi passes 73 tasks (45%) while Astra passes 101 (63%).
The takeaway: leaderboards based on "passed at least once" substantially overstate weaker models — the stability gap is far larger than the single-attempt score gap.
More from Models
- Meta ships Muse Wonder specialized models: 2.4x less forgetting, 67% better unsticking — AIatMeta · 2026-11-07
- Generated worlds tested at runtime: local interactions 83/100, stateful workflows just 30/100 — nptacek · 2026-11-07
- Steering a single neuron gives Qwen3-30B a new identity in ~1-2% of replies — Sauers_ · 2026-11-07
- OpenAI and Anthropic stuck in a release standoff as both hold back their next models — kimmonismus · 2026-11-07
- Deep Learning Weekly #480 ships with 'Gemini 4 Carbon' mention and model-routing paper — dl_weekly · 2026-11-07
- PostTrainBench: AI agents' post-training score jumps from 23% to 45% in 7 months — karinanguyen · 2026-11-07