Taste-Bench: best model flubs 40% of long-horizon agent forks, longer reasoning doesn't help

alex_verem · x · 2026-09-24

A Microsoft/City University of Hong Kong paper, The Tasteful Agent, argues long-run agent outcomes hinge on decision quality at "forks," a capability the authors call taste.

Taste-Bench

Mined from 2,677 coding and 1,132 research runs: 502 fork questions kept from 4,657 candidates, with 98.8% human agreement, no manual annotation needed.
14 frontier models tested; answers must hold under option-order swaps (chance = 25%).

Findings

Best model (GPT-5.6 Sol) scores 59.7% — still wrong on 4 of 10 critical calls.
When the deciding clue is already in context, models average 62.3%; when it only emerges after more work, they drop to 21%, below chance — exactly the kind of decision long tasks are made of.
More thinking time doesn't help: across three reasoning levels, no gains, and the most tokens were burned on the decisions models got wrong most.

Taste is trainable

A 27B Qwen3.6 trained on fork outcomes gained 17.9 points on unseen forks; advising a coding agent before 41 SWE-bench Pro tasks lifted success from 14.6% to 33.7% — more than double (perfect advice caps at 39%).

The paper directly challenges the "bigger model, longer thinking" pitch: on the decisions that sink long tasks, longer reasoning changes nothing.

Original post →

Taste-Bench: best model flubs 40% of long-horizon agent forks, longer reasoning doesn't help

More from coding & agent