Rethinking LLM Generalization: 'General Intelligence' is an Illusion of Aggregated Data

joshua_saxe · x · 2026-08-13

The author argues that the median opinion is overly bullish on LLM generalization. Early scaling papers led people to believe that scaling pre-training was hill-climbing a latent 'g' (general intelligence).

Key Points:

The Reality of Generalization: 'g' is actually weaker than expected. Generalization hinges heavily on massive pre-training corpora containing world knowledge and SFT-like examples. Scaling up models merely provides more capacity to absorb and generalize 'around' this specific data.
Current Hill-Climbing Paradigm: Today, LLMs are improved by explicitly acquiring data and RL environments for specific domains. The typical workflow is: identify a weak area → gather domain-specific data → create evals → hill-climb via better data and evals.
The Core Nature: An LLM is essentially merging many domain-specific models into a single parameter space with limited knowledge transfer between tasks. While it feels like 'g' is being hill-climbed, it is actually an aggregate of explicitly trained-in domains, each requiring an approximately linear cost in data acquisition and evals engineering.

Original post →

Rethinking LLM Generalization: 'General Intelligence' is an Illusion of Aggregated Data

More from AGI Musings