Only 16% of AI benchmarks measure organizational difficulties of real jobs
ben_burtenshaw · x · 2026-11-06
victorbarres argues that benchmark progress is often read as the answer to 'can AI do your job?', but benchmarks are made of isolated tasks while a job is done with others who hold information, make decisions and depend on you. The study finds only 16% of benchmarks measure these organizational difficulties.
More from AGI Musings
- Bank of England report: tech growth driven by company spending, not worker efficiency — GaryMarcus · 2026-11-06
- Schmidhuber: Supersmart AIs will transform the galaxy in a few hundred thousand years — SchmidhuberAI · 2026-11-06
- Musk: civilizations follow a life cycle like organisms, none lasts forever — XFreeze · 2026-11-06
- AI feeds gone quiet: are frontier labs pacing releases or out of breakthroughs? — heypearlai · 2026-11-06
- OpenAI researcher invokes 2019 double descent paper in scaling-law spat — willdepue · 2026-11-06
- A neat qualia argument: introspection can never reveal the absence of qualia — Sauers_ · 2026-11-06