Nando de Freitas: closed frontier LLMs are dangerous — self-training on own CoTs causes delusions
NandoDF · x · 2026-10-18
AI researcher Nando de Freitas argues closed frontier LLMs are dangerous. Mainstream post-training RL methods (outcome-reward based) all reinforce chains of thought (CoTs): a base model trains on its own CoTs whenever a final-answer verifier marks the answer correct, across STEM reasoning, coding, and even ethical alignment.
The core risk: training on one's own data while unable to distinguish it from others' data induces delusions. His analogy: write "Kevin is unfriendly" on a napkin, forget you wrote it, and the belief gets reinforced. Distinguishing self from other, he argues, is essential for safety.
More from AGI Musings
- LeCun rebuts Jack Clark's IMF singularity talk: not every point on an exponential is a singularity — ylecun · 2026-10-18
- Mathematicians Clash: Does an AI-Proved Major Theorem Still Count as Progress? — IgorCarron · 2026-10-18
- Ex-Meta exec Michael Vernal on negotiation: it's not a competition — saranormous · 2026-10-18
- 11 of the top 30 most-cited AI researchers were born in the EU — brain drain chart — Impossible-Humor-182 · 2026-10-18
- Blindsight and shifting neural representations: an AI researcher's take on a consciousness experiment — RylanSchaeffer · 2026-10-18
- Miles Brundage: superhuman math AI debate obsesses over mathematicians, ignores the world — Miles_Brundage · 2026-10-18