General Intelligence Index: psychometric g-factor over 59 benchmarks and 267 models
wyatt400 · reddit · 2026-09-06
The author argues that AI leaderboards averaging benchmarks have structural flaws: equal weighting of unequal tests, correlated benchmarks double-counting abilities, and rankings shifting with benchmark selection. Their General Intelligence Index (GII) adapts psychometric methods from IQ testing, asking what latent general ability best explains a model's full benchmark pattern.
Using multidimensional item-response theory over 59 benchmarks and 267 models, GII estimates a latent g factor while accounting for g-loadings, difficulty, discriminability, reliability, measurement error, and inter-benchmark covariance — discounting redundancy (five correlated coding benchmarks don't get five votes).
Estimated g-loadings range from ANLI (.986), ARC-AGI (.958), HLE (.952) down to LAMBADA (.506) and TriviaQA (.290). Scores normalize to an AIQ scale (mean 100, SD 15): current leaders are Claude Fable 5.1 and GPT-6 Astra tied at 137 AIQ, with 90% confidence intervals reported.
More from Models
- Miles Brundage asks if GPT-6 is the public name and Astra the insider codename — Miles_Brundage · 2026-09-06
- Dev wowed as Astra nails a full Three.js run and shows off Blender skills — victormustar · 2026-09-06
- Clinical benchmarks show Astra is incremental, still trailing Anthropic's frontier — danielmckinn0n · 2026-09-06
- Zvi's framework: honeypot attempt reductions up to 50-75% are fine, beyond that alarming — TheZvi · 2026-09-06
- GPT-3 to GPT-6: OpenAI API prices dropped up to 6x in five years — BLUECOW009 · 2026-09-06
- Astra called OpenAI's worst release in a while: 'severely undercooked' at coding — tokenbender · 2026-09-06