FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563
DeepLethe · hf · 2026-10-16
FDE-Gym ("The Last Mile of AI Delivery") evaluates AI agents on realistic customer delivery tasks, revealing a substantial gap versus well-specified benchmarks.
- Tasks derived from real deliveries since 2024 in China, Singapore, and the US across 18 sectors: energy, healthcare, robotics, logistics, power grid, and more
- 20 scenarios with 130+ scripted people, 95 tasks, 10,000+ expert hours; expert variants encode different unstated facts behind the same problem
- Focuses on clarifying ambiguous requirements and discovering unstated constraints, with 270+ strategies calibrated through expert review rounds
- 13 model configurations across L0–L3 difficulty. opus-5.5 leads at 0.563, sonnet-5.5 at 0.487, gpt-6-astra at 0.381; several models score negative overall (mistral-small-4 at -0.322), meaning the delivery cost the customer more than doing nothing
More from coding & agent
- Agent named 'Ilya' reportedly scores 1.1pt higher on benchmarks — daniel_mac8 · 2026-10-16
- Gemini CLI fix: substring matching inlined unrequested binary files into model context — priyanshu1976 · 2026-10-16
- Building AI agents on real healthcare claims data: the domain work that makes them work — alex_verem · 2026-10-16
- OpenAI stealth-testing mysterious internal model 'Aeon' in Codex desktop app and Dots — cedric_chee · 2026-10-16
- Vibe-coded iPhone Duo sniping bot built with Codex in 17 minutes, 41 tests passing — xiaohu · 2026-10-16
- Long-term multiplayer agent memory tool tops BEAM memory benchmark — CurieuxExplorer · 2026-10-16