Nat Lambert questions: was all that work worth just a 5-hour pre-training speedup?
natolambert · x · 2026-11-14
Nathan Lambert questions the payoff of an engineering effort that, per the referenced material, only speeds up pre-training by about 5 hours. He notes it's better for RL training and clearly better for inference, but is surprised the pre-training gain alone justified the effort. A notable commentary on the cost-benefit of pre-training optimizations versus RL and inference work.
Related event: Researchers Question Payoff of Inkling 2 Small's Native 4-bit Training(2 posts)→
More from Infra
- Jensen Huang calls TSMC chief directly over GPU batch delays, sparking org debate — IgorCarron · 2026-11-14
- NVIDIA and TSMC reportedly never signed a formal contract in nearly 30 years of partnership — IgorCarron · 2026-11-14
- SemiAnalysis: SpaceX Memphis data centers to hit 2.4 GW by end of 2026, from 0.4 GW in January — mark_k · 2026-11-14
- SemiAnalysis warns VCs may lure away NVIDIA execs with billions as AI chip gold rush heats up — elonmusk · 2026-11-14
- Tesla Optimus plus Starlink could put robots in remote sites before full autonomy arrives — jeasinema · 2026-11-14
- Running AI on e-waste GPUs: 2.5 hours to render a 15s video on dual RTX 2060s — DevGnoll · 2026-11-14