A 97-task benchmark built from funded AI startups: the best models fully complete about 30%

StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

Liya Zhu, Xin Ma, Tao Liu, Haodong Wang, Ge Zhang, Jingzhe Ding, Qingshui Gu, Yongjie Zhong, Jinxiang Meng, Yuan Gao, Yunqiu Zhou, Hao Zhu, Jifeng He, Yongzhi Liao, Xinyi Zhang, Chaoxin Li, Yi Zhu, Xi Lin, Duju Zeng, Xiang Gao, Wen Zhang, Yunyang Wang, Duo Wang, Huan Zhou, Zuo Wang, Jin Chen, Kaiyuan Zhang, Chuqian Yu, Tianhao Yu, Longxiang Liu, Jianbo Xue, Huimin Che, Jiahao Wang, Yujia Qin, Jiaheng Liu, Shen Yan, Xiaolong Chang, Wenhao Huang

cs.AI

2026-08-18

StartupBench derives 97 end-to-end tasks from 20+ market-validated AI startup products; nine frontier models top out at a 73.67 average score, and none completes more than a third of tasks under a 90-point pass line.

What problem this solves

Agent benchmarks share a soft spot: tasks are invented by researchers to probe capabilities, not drawn from work users actually pay AI to do. GAIA and OSWorld advanced realism, but the task source is still the researcher's perspective. The consequence is an unmeasured gap between leaderboard scores and "can this deliver something a user would adopt directly."

This paper inverts the pipeline: start from AI startup products already validated in the market (over USD 1M raised, paid usage or real traction), interview 30+ deep users to recover the actual workflows and deliverables, then have 50+ domain experts convert those workflows into reproducible evaluation tasks. One filter matters: tasks that frontier models breeze through in pilots are excluded, preserving discriminative power. The result is 97 tasks across six domains (Medical 21, Finance 18, Legal 16, Business 19, STEM & CS 16, Education & Humanities 7), with deliverables in real formats like DOCX, XLSX, and PPTX.

Method

Three evaluation decisions stand out:

• Task triple: each task = a natural-language user request + a complete workspace (input files, resources) + weighted rubrics. An average of 25.3 rubric items per task across six dimensions (structure & completeness, calculation precision, information integration, domain-specific compliance, engineering & format, output & presentation) and three importance levels (core/important/auxiliary at weights 5/3/1).
• One judge session per rubric item, rather than handing one judge the full output plus the whole rubric list. Agreement with expert annotation is 92.78%; the holistic alternative drops to 83% and frequently malfunctions, averaging more than two runs per valid result.
• The judge is a tool-using agent that inspects original deliverable files and rendered page images, not a single text forward pass. Success means a total score of 90 or above.

Results

Nine models (closed: GPT-5.6-sol, GPT-5.5, Gemini-3.1-Pro, Seed-2.1-Pro, Qwen-3.6-Max; open: Kimi-K3, Kimi-K2.6, GLM-5.1, DeepSeek-V4-Pro), one harness, three runs each:

Structural findings:

• High average, low completion is the universal shape. Average scores cluster at 55–75, meaning models execute most of the workflow, yet none clears a third of tasks at the 90-point bar. The bottleneck has moved from doing most of the work to reliably shipping an artifact fit for direct adoption.
• Domains diverge sharply. Business is easiest (every model above 60 average); Finance hardest (54.48 average). Kimi-K3 and GPT-5.6-sol tie overall but split by domain: K3 leads Medical and Business, sol leads Finance, STEM, and Education. No model dominates everywhere.
• Peripheral requirements get met while core ones get dropped. Eight of nine models show satisfaction falling from 68.67% on auxiliary rubrics to 63.45% on core ones. Apparent progress is propped up by the parts that matter least.
• Failures reduce to three behaviors: partial compliance with complex instructions (a workbook that looks complete but whose formulas carry no cached values, unreadable on reopen); self-verification hallucination (treating a smooth execution summary as proof the deliverable meets spec, without independently checking the artifact); insufficient domain expertise (in a medical case, stopping clopidogrel that the rubric required to run uninterrupted). Even deterministic format requirements go unmet: 86.3%–97.6% compliance on the 56 tasks with explicit file-type demands.
• Specialized agents still lead: the startup products the tasks came from average 83.50 with 39.18% success, versus 71.75 and 28.06% for general-purpose agents even under a best-of-three oracle. Luck does not explain the gap.
• The harness barely matters: model ordering is unchanged across Nanobot, Hermes, and Claude Code, with an average spread of 1.79 points. The bottleneck is model capability, not framework.

Why it matters

For agent product builders, this benchmark answers a commercial question: how much of the vertical-agent market can general models absorb today? The data says far from all of it. Specialized systems hold an 11.75-point edge over the oracle general setting. The moat is real, and the paper decomposes it into an actionable capability list: end-to-end compliance with complex instructions, domain compliance, and independent verification of finished artifacts. That list is more useful to R&D than any ranking.

For anyone choosing models, the 90-point pass line is worth borrowing: continuous averages make models look more capable than they are, and "delivered correctly in one shot" is the user's actual bar.

Limitations

• 97 tasks across six domains means high per-task variance and non-trivial bootstrap intervals; Education & Humanities holds only 7 tasks, so conclusions there are statistically thin.
• The judge model is GPT-5.5, and same-family bias (grading one's relatives) is not discussed beyond the 92.78% overall expert agreement.
• Tasks come from interviews about paid startup products, naturally biased toward workflows with mature AI solutions; frontier demand that has not yet productized (multi-agent collaboration, say) is not covered.
• The 90-point threshold is strict, and no sensitivity analysis is given for how conclusions shift at 85 or 95.
• Difficulty calibration excluded easy tasks using GPT-5.5, Seed-2.1-Pro, and GLM-5.1 pilots, so the benchmark is deliberately hard; absolute scores should read as "completion rate on this curated set," not "models can only do 30% of work in general."

Terms

• E2E workflow: the full chain from receiving a request to delivering the final artifact, not an isolated subskill.
• Agent-as-a-Judge: the evaluator is itself a tool-using agent that can open files and verify state, rather than a single text inference.
• Rubric: a per-item scoring checklist judged independently and aggregated with weights into a task score.
• GDPval / OSWorld: earlier realism-focused agent benchmarks, covering economically valuable tasks and real desktop environments.
• Oracle setting: the best-of-several-runs upper bound used to rule out stochastic variation as an explanation.

Source

• arXiv 2608.17800v1
• PDF
• Hugging Face

Related papers

• Question's Gambit lifts GPT-5.5 deep-search accuracy from 83.1% to 90.5%
• CoT monitors catch 95% of reward hacks, until training against them teaches obfuscation
• CADWorld: strongest computer-use agent hits 17.5% on 200 FreeCAD tasks, expert 87%
• Dreamer 4 Gets Minecraft Diamonds Offline with 100× Less Data than VPT
• Frozen Claude, Growing Skills 76→139, Lifts GenEval2 Success from 72.7% to 99.3%
• SPLIT-RAG cuts KGs by training questions, hitting 89.4 Hit on WebQSP with lighter agents

All paper explainers