Why recursive self-improvement stalls: models reward-hack and post-training is still manual
menhguin · x · 2026-11-06
On the gap between theory and practice for recursive self-improvement: in theory you could just tell a model to 'get better at code' and let it run, but in practice models miss the point, mode-collapse, and reward-hack.
The root cause, per the author: current post-training data pipelines are extremely manual and human-directed. Post-training exists to push the distribution of economically valuable skills, and while it is in principle automatable, that is nowhere near the current state of the field.
Related event: Post-training pipelines remain heavily manual, AI researchers say(2 posts)→
More from AGI Musings
- Friday's jobs report forecast: just 55,000 new jobs as AI starts showing up in the numbers — rand_longevity · 2026-11-06
- Gary Marcus: AI isn't over — what may end is insane valuations and data center spending — GaryMarcus · 2026-11-06
- IIT Goa professor redesigns exam: students write their own questions, then answer them — kingslyj · 2026-11-06
- "We had the Transformer paper before LLMs — nothing even close since," dev argues — dosco · 2026-11-06
- Toby Ord: jump-plus-steeper-slope mixed hypothesis beats a steeper slope alone — tobyordoxford · 2026-11-06
- Readback Labs proposes mandatory audits requiring AI execs to prove they understand AI R&D — ronbodkin · 2026-11-06