AI agent misses $14M in lease clauses: Surge AI's GDP.xlsx benchmarks agents on real spreadsheets
echen · x · 2026-10-10
Surge AI released GDP.xlsx, a benchmark of 70 real-world tasks spanning 12 domains (finance, real estate, legal, etc.) testing whether frontier models can work with professional Excel workbooks. The insight: a real spreadsheet is closer to a small software system than a table — the hard part is reconstructing workbook logic, knowing which sheet holds the truth, and decoding rules encoded in colors, comments, and formulas. In one case study, an agent found every expiring rent discount in a commercial real estate workbook but missed three break clauses worth roughly $14 million in forecast rent: the task required following two paths through a 15-tab workbook, and the model followed one and stopped. Knowing when you're done, Surge argues, is part of the job.
More from coding & agent
- Testing an AI agent to book a hotel failed — and it still required manually entering credit card info — NielsRogge · 2026-10-10
- AI now writes 52% of merged code, but DX researcher says stop treating AICP as a KPI — rseroter · 2026-10-10
- OpenAI says Codex usage data will never power its predictions, even after testing — btibor91 · 2026-10-10
- Summarize: free Mac tool summarizes anything on screen in ~1s via Cerebras, one command to install — draecomino · 2026-10-10
- Building Blender Scenes Entirely With AI, Without Ever Touching Blender — RexDouglass · 2026-10-10
- More details on the Jev Router for model routing workflows — omarsar0 · 2026-10-10