ArchitectureQ benchmark: frontier LLMs hit 76% on training intuition vs 66% top humans
gregmushen · x · 2026-10-13
The ArchitectureQ benchmark tests whether LLMs can predict which training recipe works best. Findings: frontier models reach 76% accuracy (random 33%) vs 66% for the best human researcher, yet are imperfect — on architecture-only questions the best human hits 65% while GPT-6 Astra gets 38%; intuition is empirical not structured (CoT doesn't help); a 20-item knowledge base lets GPT-4o nearly match Claude Opus 5; and models are insensitive to dataset properties, suggesting data is AI's real 'dark matter'.
More from Models
- Ex-OpenAI VP Brundage: Meta Maximum subscribers can't tell if they're using Wonder Full — Miles_Brundage · 2026-10-13
- Unverified: Meta rumored to launch Muse Wonder agentic model series — scaling01 · 2026-10-13
- FrontierCode 1.1 leaderboard: Claude Opus 5.5 tops at 52% as leading models fail 40%+ of tasks — ArtificialAnlys · 2026-10-13
- Alexandr Wang hails Meta's Muse as benchmark-competitive, predicts consumer agent landslide — alexandr_wang · 2026-10-13
- LiveBench 2.0 incoming: agentic long-horizon benchmark you can't benchmaxx — bindureddy · 2026-10-13
- Muse Wonder shows far better OOD generalization than Muse Spark 1.3 — armand_ruiz · 2026-10-13