FDE-Gym benchmark: best agent fully solves only 14 of 95 real customer-delivery tasks
dair_ai · x · 2026-10-16
FDE-Gym tests agents as forward-deployed engineers on 95 tasks from real delivery projects across 18 sectors, with scripted stakeholders and production-like systems. Scoring rewards improving the customer's own success metric while penalizing extra messages, trial runs and waiting time. Across 13 configurations from 7 providers, the top mean score is 0.56 and only 14 tasks are fully resolved. The main failure mode: missing unstated constraints. Scores also collapse as task instructions become less explicit (L0→L3).
Related event: FDE-Gym Benchmark Exposes AI Agents' Last-Mile Delivery Gap(2 posts)→
More from coding & agent
- Satire: 19 Claude agents 'reverse-engineer' Blender, forgetting it's already open source — Dan_Jeffries1 · 2026-10-16
- Engineers Should Mine Gong Calls for Ideas That Never Became Tickets — curious_vii · 2026-10-16
- One 80-Word Prompt, Overnight AI Agent Built a Site With 205 Bakeries Mapped — jamesbrooksco · 2026-10-16
- Visual Lean: drag-and-drop GUI for the Natural Numbers Game runs Lean 4 in WASM — MilesCranmer · 2026-10-16
- Fine-Tuned 0.6B Router Beats Its 27B Sibling: Small Specialized Models Win — mshubham · 2026-10-16
- Engineering Leader: AI Got Me Coding Again — I Built Our Feature-Flag System With an Agent — rseroter · 2026-10-16