View: Pretraining Far From Saturated; Scale Not Necessary for Peak Intelligence
kalomaze · x · 2026-08-21
Twitter user kalomaze discussed the sources of model intelligence in a reply:
- Pretraining Potential: The latent capabilities within the oldest dsv3 (DeepSeek-V3) pretraining checkpoints are far from saturated. One could freeze pretraining for 5 years, and the outcomes from unlocking that potential would still be unrecognizable.
- Model Scale: Referencing the 0731 ARC-AGI-2 score, the 0813 performance, and professor's views, model scale is argued not to be a necessary condition for peak intelligence.
- Post-training Space: While stronger pretrains are valuable for raw 'g', the space of tractable process goals (pg objectives) is incomprehensibly wide. Current efforts are only tapping the lowest-hanging fruits like rubrics and RLVR.
More from AGI Musings
- AI Is Forcing Everyone Into Management—Until It Eats Middle Management Too — MonkZer0 · 2026-10-16
- Shogun 2 remaster review-bombed over AI-upscaled graphics, sparking dev debate — draginol · 2026-10-16
- Pedro Domingos: Everyone must have their own unique model, anything else is dangerous — pmddomingos · 2026-10-16
- Interp researcher pushes back: if you can't simulate a neuron, you haven't explained it — nabla_theta · 2026-10-16
- Will AI Discoveries Outpace Human Understanding? A 'Roadside Picnic' Analogy — infinitely_zero · 2026-10-16
- Virtue ethics meets AI alignment: what would it mean to make a model virtuous? — joelbot3000 · 2026-10-16