Microsoft's Taste-Bench: best frontier model scores only 59.7% on long-horizon agent decisions
microsoft · hf · 2026-09-23
Microsoft introduces "taste" — an agent's ability to make good decisions at critical forks in long-horizon tasks (which hypothesis to test, which implementation to build on) — and argues no existing benchmark measures it.
Taste-Bench
- Automatically mines decision forks from agent trajectories in engineering and research tasks, using both parallel attempts at the same task and detours within a single trajectory — no human annotation needed
- Each question presents a fork; the model picks a direction without seeing what follows
Findings
- The best frontier model answers only 59.7% correctly
- Forks whose deciding evidence appears later in the trajectory are much harder for all models
- Larger reasoning budgets don't improve accuracy
Taste is trainable: distilling outcome-aware teacher judgment into a student model improves decisions on unseen tasks and lifts end-to-end success on held-out SWE-bench Pro tasks.
More from coding & agent
- Dev uses Opus 5.5 with Lean to formally verify Claude Agent SDK, shipping 16 bug-fix PRs — jimmykoppel · 2026-09-23
- Rogo CEO names memory compaction as the key unsolved problem for enterprise agents — rohanpaul_ai · 2026-09-23
- Graph Engineering: Building Reliable AI Agent Systems as Explicit Task Graphs — Pavan_Belagatti · 2026-09-23
- Have Your Coding Agent Attach Flame Graphs to Every PR It Opens — DanielLockyer · 2026-09-23
- Early hands-on: Sol 6 shows strength at goal-driven tasks, dev lets it run overnight — gregmushen · 2026-09-23
- Theorem says Lean-verified AI sandboxes are months away, at 1-30KB of proofs verified per hour — ctjlewis · 2026-09-23