Brute-force compute will crack every benchmark — real progress should be measured by discoveries
_galyo · x · 2026-09-05
Galyo argues: any task/environment you can reliably score and brute-force with enough money will be solved; every popular benchmark will meet this bar; abilities won't necessarily transfer. For measuring progress, he distinguishes throwaway benchmarks (for knowing the best model now) from discovery — systems doing something genuinely new and valuable, like solving open problems — as the sparser but truer signal.
More from AGI Musings
- Nat Lambert: AI cyber risk is an ecosystem issue, not an open-weights problem — natolambert · 2026-10-21
- Obesity drugs still out-earn OpenAI and Anthropic combined at $71B a year — HealthcareAIGuy · 2026-10-21
- Is an AI's 'self' just a simulation in the scaffolding layer above the model? — gerardsans · 2026-10-21
- AI can simulate how a self talks, but not what a self is — gerardsans · 2026-10-21
- Musk predicts money will become irrelevant, replaced by 'wattage and tonnage' — Polymarket · 2026-10-21
- zetalyrae quips: post-singularity utopia is so great you need brain surgery to cope — wfithian · 2026-10-21