GPT-6.1 Terra ranks #14 on Toloka Arena as pass^5 falls to 59% from ~72%
MParakhin · x · 2026-11-01
Toloka AI Arena's agentic leaderboard (τ-bench-style simulated customer-service tasks with live databases and strict business rules) uses pass^5 — the probability all 5 independent trials succeed. OpenAI's latest GPT-6.1 Terra lands #14 of 34 at 59.0 pass^5, down from 72% single-run, revealing inconsistency: it reaches the right outcome but skips or reorders required steps — fine in demos, failing audits. Former Microsoft AI chief Mikhail Parakhin adds practical advice: larger models and higher reasoning effort are more consistent at higher cost, and he runs 4x critics in a critique loop to lower bad misses.
More from coding & agent
- 1,500 Agent Sessions Show Tool I/O Eats 57% of Context While Dialogue Is Just 2% — rajistics · 2026-11-01
- Enterprise RAG: fixing retrieval first cuts failed retrievals by 67% before touching the model — LangChain · 2026-11-01
- RSIGame self-improvement loops lift Godot game quality from 50.26 to 64.53 — jiqizhixin · 2026-11-01
- Trishool's HaloGuard hits 83.63 F1 SOTA for agent output guarding with a 0.8B model — markjeffrey · 2026-11-01
- GitHub makes Copilot cloud agents GA: one session across web, mobile, CLI, Slack and Teams — 0xkarasy · 2026-11-01
- shadcn predicts every AI harness will build two UIs: Slack-style chat, then Linear-style tasks — shadcn · 2026-11-01