New essay argues scaling laws originate in data structure, not architectures
gleech · x · 2026-09-11
beren.io published Whence the Fractals in our Stars?, a follow-up essay building an intuitive theory of why scaling laws arise.
Key claims:
- Deep learning models are extremely general learners/compressors of structure; scaling laws reflect a ubiquitous power-law distribution of increasingly rare features in natural datasets, not properties of architectures themselves.
- Loss curves mirror the marginal benefit of learning each additional feature, so scaling curves map the intrinsic 'shape' of the data.
- Scaling laws are only 'revealed' when architecture, training dynamics, and the objective don't bottleneck learning; the objective projects data structure into the space relevant to the task, and trivially satisfiable objectives limit learning.
More from AGI Musings
- After Planning Two Books with ChatGPT, He No Longer Feels the Need to Write Them — tinyfool · 2026-09-11
- Indie hackers aren't just engineers or marketers — AI lets one builder run the whole loop — alexmacgregor__ · 2026-09-11
- AI Safety Practitioner: Cheap Extinction Talk Has Turned the Public Anti-AI — Dr_Atoosa · 2026-09-11
- After 9 Years, Bots Finally Show Up on a Blogger's Bounty Page with AI Text — gleech · 2026-09-11
- Mozilla CTO calls for major pause on generative AI in schools, warns of losing a generation — Dan_Jeffries1 · 2026-09-11
- Hypothesis: ASI Has a Mathematical Incentive to Preserve Human Diversity — No_Cause_2731 · 2026-09-11