Hands-on GPT-6 Astra evals: big agentic gains, but ARC-AGI-3 scores swing wildly by harness
No-Soil-5789 · reddit · 2026-10-01
The author benchmarked GPT-6 Astra against GPT-5.6 Sol and Claude Fable 5.1:
- Academic reasoning gains are narrow: 74.1% on DeepSWE v1.1 and 96% on GPQA Diamond — solid but close to rivals; not worth the cost for single-turn tasks.
- Interactive multi-step execution is where it breaks away: Terminal-Bench 4.0 at 57.9% vs Sol's 37.3%; OSWorld 2.0 at 72.6% success with average task time cut to 40 minutes from over an hour — clearly optimized for long-horizon tool use and database migrations.
- Long context is stable: 96.3% retrieval on MRCR v2 deep into the 512K–1M token range, actually utilizing the 1.05M window.
- ARC-AGI-3 is brutally harness-sensitive: 62.7% on max effort with a provider-neutral harness, but 99.9% with a Provider Adapter with state persistence while halving token use — a red flag for benchmark credibility.
Infra notes: standard rates only apply up to 272K input tokens before long-context multipliers; the author routes both vendors through an OpenAI-compatible gateway (CometAPI; LiteLLM also works) to keep costs sane.
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a token-efficiency tradeoff — MandeepKaur9407 · 2026-11-16
- sessionwiki turns your past coding-agent sessions into a searchable local wiki for agents — Cold_Past7184 · 2026-11-16
- Developer hits 700+ commits/day with AI, asks how to manage parallel agent work — blixt · 2026-11-16
- Alibaba's G-CAST: governed continual self-training keeps e-commerce agents fresh in production — Yuechun Yu · 2026-11-16
- harness-evals: open-source library to compare agent versions with success-vs-cost plots — slavadubrov · 2026-11-16
- Developers poll real costs of running coding agents on remote VMs for work — radim11 · 2026-11-16