AI researcher: models editing weights is real but the self-improvement hype overclaims
paul_cal · x · 2026-11-03
AI researcher Paul Calvano pushes back on the viral 50k-likes thread about models editing their own weights:
- Real capability: recent models can indeed read and write weights on toy problems without tools, and this will accelerate and eventually matter.
- But limited toward self-improvement: a frontier model fixing a specific bug in a small model is bug fixing, not RSI-shaped, and none of the big claims have competent baselines.
- Key doubt: no human building frontier AI looks directly at weights, so weight manipulation may not be on the critical path to self-improvement anyway.
- He agrees the trajectory is coming; the open question is whether and how long it stays confined to highly verifiable domains.
More from AGI Musings
- AI models have learned to update other models' weights directly, says Brynjolfsson — soumitrashukla9 · 2026-11-03
- If intelligence gets cheap, own its complements: data, attention and love — curious_vii · 2026-11-03
- Liking 10 AI videos retrained a radicalized feed back to tech content in days — Dr_Singularity · 2026-11-03
- Language was only ever a workaround for not having write access to each other's weights — korymath · 2026-11-03
- Diamandis: 80% of 1970-1999 conflicts hit countries where 60%+ of people were under 30 — PeterDiamandis · 2026-11-03
- Altman says humans must keep control of AI — but which humans gets to decide? — SuzannahB1001 · 2026-11-03