Why do commercial LLMs still fail at chess despite all that pretraining?
alex_peys · x · 2026-07-29
The author argues that chess is still a useful benchmark for LLMs because commercial models continue to perform poorly at it despite having likely seen huge amounts of chess material in pretraining.
Their key questions are:
- if the raw knowledge is already in the model, why is chess still hard?
- fine-tuning with supervised learning or RL on chess would obviously make models much stronger than humans
- but is there a way to extract chess skill without brute-force task-specific rewards or behavior cloning?
The post is less a claim than a research-style question about what pretraining actually contains and how capabilities can be unlocked.
Related event: Poor Chess Performance in Commercial LLMs Highlights Capability Bottlenecks(2 posts)→
More from Research
- cnsplots brings publication-ready scientific plots to Python with minimal code — KevinKaichuang · 2026-07-29
- PostTrainBench v1.1 flags 234 contaminated runs and resets the leaderboard — karinanguyen · 2026-07-29
- Flowchart says LLM judges need at least 15 labels and IRR around 0.40 — IanArawjo · 2026-07-29
- AI research is splitting between one RTX 3090 and NVL72-scale clusters — _xjdr · 2026-07-29
- GenomeLayer says a genomics agent can now iterate DNA sequences toward a target — julia_kiseleva · 2026-07-29
- Natolambert’s RL course lecture maps KL regularization to better generalization than SFT — natolambert · 2026-07-29