Claude Opus gains 250 Elo playing 200 chess games, while GPT-6 Astra drops 410
新智元 · wechat · 2026-10-05
Peter Gostev, AI capability lead at Arena, ran an experiment letting models play 200 chess games against Stockfish with no weight updates — only self-written notes for memory.
- Setup: three Stockfish tiers (Skill0, capped 1800, full strength), models choose difficulty; GPT ran via Codex, Claude via Claude Code; engine use banned and verified. Inspired by Reddit's "time loop" thought experiment.
- Opus 5.5 improved: stuck at 1500 for 75 games, then surged to a peak of 1840, settling at 1760 by game 194; win rate vs the 1800 tier rose from 30% to 50%. Oriol Vinyals called it textbook in-context learning.
- GPT-6 Astra declined: peaked at 1810 by game 29, then fell to 1400; score rate vs 1800 tier dropped from 44% to 12%.
- Hypothesis: Gostev suspects Astra's 272k context window degraded as notes piled up — unproven, so he spun up an 828k-context GPT-6.1 Sol track to test it.
- Elo is estimated from recent 50 games vs weaker tiers, not a human-equivalent rating.
The experiment probes whether frozen-weight models can self-improve through in-context notes — Opus's +250 suggests yes.
More from Models
- Developer slams Grok models as 'garbage', shares coding agent's repetitive trial-and-error log — feregri_no · 2026-11-20
- Drex Router cuts small-game build cost to $1.26 vs $13.44 on Claude Opus alone — testingcatalog · 2026-11-20
- OpenAI slammed for AI slop: 'Just hire photographers to RLHF imagegen for a day' — flowersslop · 2026-11-20
- Model Fight Card 16: Tenet models dominate jailbreak arena with 21/24 escapes — Dave_Maynor · 2026-11-20
- Coding agent index shows max effort costs 6x for fewer points; Gemini 4 Neon wins on value — Arindam_1729 · 2026-11-20
- Reddit: Claude Max for $29 Instead of $200 — How Do Resellers Profit? — mehdiweb · 2026-11-20