Continuous learning benchmark has models learn chess over 200 games — Elo barely improves
imjustnewatai · x · 2026-10-04
Peter Gostev built a "continuous learning" benchmark where models are given the goal of learning to play chess against a Stockfish opponent over 200 games — they can pick difficulty and take notes, but no cheating via engines. A live site tracks their Elo journey.
Early results:
- GPT-6 Astra: finished its 200 games with a negative Elo trend (vs full-strength Stockfish: 66 losses, 2 draws in 68 games, rolling score 0.01)
- Opus: a slight positive improvement, possibly within random noise; still playing
The benchmark probes whether models can self-improve without a real continuous-learning mechanism — an early proxy for RSI. The retweeter says he wants to build something similar.
More from Models
- Microsoft's MAI models have been front-page on Hacker News since Ignite debut — marlene_zw · 2026-11-20
- Gemini 4 Carbon tops graduate-level physics benchmark but loses on agentic tasks — ArtificialAnlys · 2026-11-20
- Neon and Hydrogen own every Pareto frontier in Gemini 4 lineup, from $0.29 to $6.44 per task — ArtificialAnlys · 2026-11-20
- Gemini 4 lineup pricing revealed: Carbon $4/$20, Neon $3/$15, Hydrogen $1.50/$7.50 — ArtificialAnlys · 2026-11-20
- Google opens Project Genie 4 waitlist access to Google AI Pro subscribers — testingcatalog · 2026-11-20
- Unverified: Grok 4.8 quietly deployed in Grok Bot, official release expected next week — mark_k · 2026-11-20