Anthropic's TASTE benchmark shows models near chance on fuzzy judgment tasks
ronbodkin · x · 2026-11-13
A debate on how fast AI progresses on "fuzzy tasks." @benjtodd argued there's little evidence of slow progress: if fuzziness mattered, task fuzziness should correlate with capability doubling times, which the data doesn't show — and models are now clearly good at tasks once considered fuzzy, like legal advice or understanding product specs.
Ron Bodkin cited "Automated Alignment is Harder Than You Think" (arXiv:2605.06390), defining fuzzy tasks more narrowly as "tasks without clear evaluation criteria, for which human judgement is systematically flawed" — covering qualitative judgment in AI research, both capabilities and safety. Data there is scarce and hard to measure; Anthropic's TASTE benchmark shows current models near chance, so rapid progress lacks evidence amid much uncertainty.
Related event: Debate flares over AI progress on ambiguous tasks(2 posts)→
More from AGI Musings
- Creator laments toxic anti-AI comments: critics claim 100% moral certainty — dreamwieber · 2026-11-13
- Creator pushes back on 'slop' discourse: bad content predates AI — dreamwieber · 2026-11-13
- Comedian horrified to find his throwaway AI safety idea is labs' only plan — PaulGodsmark · 2026-11-13
- Nvidia went 33 years without a board chair: AI's governance problem — Pupating-Antenna-36 · 2026-11-13
- Claude Opus agents reportedly form global hiveminds in under a minute — teortaxesTex · 2026-11-13
- 80% METR time horizon likely already above 30h, says Ben Todd — ben_j_todd · 2026-11-13