TASTE benchmark: nearly all frontier models score within noise of random chance
maksym_andr · x · 2026-11-14
A TASTE paper figure shows nearly every frontier model tested — Anthropic's Opus/Haiku/Sonnet lineup, Gemini, Qwen, GLM, Kimi, Grok, and all GPT-5.6 variants — scoring within the 95% binomial noise band of random chance (50%), vs. 77% expert human performance; only Fable 5 edges barely above chance. The author guesses this Benchmark 14 measures something like research taste, and notes it's the one benchmark showing no progress across all models.
Related event: Anthropic's TASTE Benchmark Finds Models Poor at Judging AI Safety Research(3 posts)→
More from Models
- Together AI webinar dissects Ripple-65B coding model: dense vs MoE, ablations, RL runs — togethercompute · 2026-11-14
- Skeptic doubts Anthropic achieved large-factor bpb gains at fixed compute — teortaxesTex · 2026-11-14
- Surge AI benchmark: every Claude model, even Haiku, beats every GPT on staff-level coding — echen · 2026-11-14
- AI cracks Gödel's 1,009-page notebooks, uncovering hundreds of unknown theorems in days — stevenstrogatz · 2026-11-14
- NVIDIA lands nvfp4 in large-scale pretraining under codename "yolo bundle" — SonglinYang4 · 2026-11-14
- Delip Rao mocks vendors testing safety in production, then rolling back updates — deliprao · 2026-11-14