Taste-Bench: best model flubs 40% of long-horizon agent forks, longer reasoning doesn't help
alex_verem · x · 2026-09-24
A Microsoft/City University of Hong Kong paper, The Tasteful Agent, argues long-run agent outcomes hinge on decision quality at "forks," a capability the authors call taste.
Taste-Bench
- Mined from 2,677 coding and 1,132 research runs: 502 fork questions kept from 4,657 candidates, with 98.8% human agreement, no manual annotation needed.
- 14 frontier models tested; answers must hold under option-order swaps (chance = 25%).
Findings
- Best model (GPT-5.6 Sol) scores 59.7% — still wrong on 4 of 10 critical calls.
- When the deciding clue is already in context, models average 62.3%; when it only emerges after more work, they drop to 21%, below chance — exactly the kind of decision long tasks are made of.
- More thinking time doesn't help: across three reasoning levels, no gains, and the most tokens were burned on the decisions models got wrong most.
Taste is trainable
A 27B Qwen3.6 trained on fork outcomes gained 17.9 points on unseen forks; advising a coding agent before 41 SWE-bench Pro tasks lifted success from 14.6% to 33.7% — more than double (perfect advice caps at 39%).
The paper directly challenges the "bigger model, longer thinking" pitch: on the decisions that sink long tasks, longer reasoning changes nothing.
Related event: Microsoft's Taste-Bench Shows Top Models Fail 40% of Long-Horizon Decisions(2 posts)→
More from coding & agent
- YC-backed Roma turns your to-do list into an agent command center across Gmail and Notion — testingcatalog · 2026-11-12
- Boubou: open-source CLI turns your laptop into a fully local voice AI hub — alexcovo_eth · 2026-11-12
- User automates entire video clipping workflow with Nous Research's Hermes agent — Teknium · 2026-11-12
- Tool-call failure rates jump 14x across providers, charts pinpoint inference stack updates — remilouf · 2026-11-12
- As models improve, writing good evals gets harder and more important — amankhan · 2026-11-12
- Prime agent ships with day-one multi-agent communication, runs in Minecraft unmodified — sarahcat21 · 2026-11-12