Researchers Debate Measuring AI Progress: Report Efficiency Gains, Not Final Benchmarks
datagenproc · x · 2026-11-12
A debate between datagenproc and eliebakouch on attributing AI progress: final benchmark scores are biased by training compute, so efficiency-gain metrics should be reported; attribution also needs to account for experiment compute, headcount and data growth.
More from AGI Musings
- Amateurs with Claude + Codex claim discovery of a new dwarf planet in public telescope data — willdepue · 2026-11-12
- Adam Scholl: strong evidence AI is useful, weak evidence it generalizes — gleech · 2026-11-12
- "SF is right long-term, NYC right near-term, nobody has a clue mid-term" on AI — austinc3301 · 2026-11-12
- Andrew Critch: stop calling extinction risks "improvement"—RSM vs RSI matters — AndrewCritchPhD · 2026-11-12
- Canada tech vacancies down 46.9% from peak; software engineer roles fell 72% — RichardsonDx · 2026-11-12
- Any US State Could Legalize AI Safety Coordination via Parker Immunity — TinfoilTricorn · 2026-11-12