Architecture IQ benchmark shows LLMs still lag humans in training intuition, ResNet-style

ZimingLiu11 · x · 2026-10-24

Ziming Liu (KAN author) introduces Architecture IQ, a benchmark measuring LLM "training intuition" — predicting which model/optimizer/hyperparameters yield the lowest test loss. Key findings: models rank optimizers well (85–87%) but struggle with architectures (38–61%); most repeat their pick when dataset flips the winner, at or below the 50% data-blind bound; more reasoning tokens bring no consistent gain. An Opus-5-built evidence-weighted knowledge base of up to 20 training-physics principles lifts GPT-4o from 53.2% to 67.6% on 50 held-out questions without weight updates. The framing: He Kaiming created ResNet by intuition without 1M-token CoTs — humans still surpass LLMs in training intuition, in some cases.

Original post →

Architecture IQ benchmark shows LLMs still lag humans in training intuition, ResNet-style

More from Models