Mercor's 'organizational difficulty' taxonomy: only 16% of AI benchmarks measure org skills
_jasonwei · x · 2026-11-07
Mercor, with Copenhagen Business School professors, argues 'a task ≠ a job': benchmarks measure isolated tasks while most valuable work happens inside organizations. Their new taxonomy decomposes 'organizational difficulty' into dimensions like navigating authority structures and task interdependence. Strikingly, fewer than a quarter of prominent AI benchmarks even claim to represent organizational elements, and only 16% separately report model performance on them. The goal: an ONET-like organizational counterpart so evals can place models inside an org chart.
Related event: Only 16% of AI Benchmarks Capture Organizational Difficulty, Study Finds(2 posts)→
More from AGI Musings
- AI Exposes the Economy's Inefficiency: The Real Threat Is Getting Paid for Mediocrity — TinfoilTricorn · 2026-11-07
- basedjensen: recursive self-improvement is already here and bans are unenforceable — basedjensen · 2026-11-07
- Musing: Much of ML's progress may just be rediscovering good discipline — yunta_tsai · 2026-11-07
- Geoffrey Hinton's 47-minute AI masterclass promises to reshape how you think about AI — HeyAmit_ · 2026-11-07
- AI video characters may feel conscious sooner than LLMs, argues viral post — flowersslop · 2026-11-07
- WIRED archive analysis: 60.5% of AI stories use harm framing vs 18.6% in early internet era — emollick · 2026-11-07