Pretraining vs RL debate is silly — evidence says you need both, argues voooooogel
voooooogel · x · 2026-11-21
Reacting to the "is pretraining or RL more important" debate, voooooogel calls it a chicken-and-egg argument. His case: we have good evidence pretraining alone isn't enough (405B base hasn't proved anything), and very strong evidence that pretraining + RL works. If someone wants to claim LLMs are hitting a wall because the pretraining base or "convex hull of human ideas" is exhausted, they should say so directly — phrased that way, it sounds like plenty of bets that didn't age well. The question of which component is "more important" is moot; whether pretraining gains have truly plateaued is the real debate.
More from AGI Musings
- Vishal Sikka on reported Riemann Hypothesis proof with Claude: 'an incredible day' — vsikka · 2026-11-21
- Tamay offers 2:1 odds that RH won't be solved within two years — panickssery · 2026-11-21
- Ex-Tesla AI Chief Predicts Labs Won't Need New Pretraining Researchers by 2027 — jasondeanlee · 2026-11-21
- Dev Laments: AI Coding Offloaded the Fun Parts, Leaving Only Reviewing and Correcting — pritisinghhhh · 2026-11-21
- Chollet: AI made staggering leaps since 2024, yet NVDA buyers are down 5% — fchollet · 2026-11-21
- Independent mathematicians attest AI Riemann hypothesis proof is almost certainly true — soumitrashukla9 · 2026-11-21