200 chess games suggest in-context learning doesn't work yet, experimenter says
akbirthko · x · 2026-10-16
The original author had models play 200 chess games (native harness, continuous context/files), then ran an Astra vs Opus tournament, concluding that in-context learning currently doesn't work. He open-sourced 551,684 records and the harness structure, with observations:
- Opus's Elo rose a lot and it ended above Astra, yet Opus then lost the match 5.5–24.5 — "a disaster"
- Models left themselves notes and learnings but still made the same mistakes
- Any Elo gain is closer to reward hacking than actual learning
- Elo drops for some models are hard to explain, but not due to limited context
A critic (akbirthko) argued that without gradient signal, what tokens-plus-context can teach is minuscule compared to the value function learned in pretraining, and that practical ICL/CL is more about messy state management and building better tools for oneself and other agents.
Related event: 200 Chess Games Test Suggests In-Context Learning Falls Short(2 posts)→
More from Models
- Gemini 4 model card: model eliminated months of trial-and-error on bioweapons research — Puzzleheaded-King584 · 2026-10-16
- Every AI picks 7: Anthropic engineer's rule for escaping model taste convergence — rubenhassid · 2026-10-16
- Brain teaser: strip all attention, MLP and embedding matrices from a Llama 3 — how many trainable params remain? — Sauers_ · 2026-10-16
- Lean v4.35.0 ships verified kernels — one written and proven consistent by Claude — geoffreyirving · 2026-10-16
- One flagged seat froze an entire Claude team org — founders share backup plans — RexDouglass · 2026-10-16
- Kimi K3 checkpoint impresses: strong held-out results point to solid pretraining — SonglinYang4 · 2026-10-16