Deepnote open-sources synthetic company to benchmark proactive agents across 12 frontier models
heyshrutimishra · x · 2026-11-13
Deepnote Research released the Agent Initiative Benchmark: they built and open-sourced a synthetic company, PageMantle — complete with CRM, product analytics, billing, a warehouse, an issue tracker and docs — then let 12 frontier models lurk inside it across 50 workplace situations to test proactive help: gathering evidence from systems the conversation never mentions, deciding whether to intervene, and routing help to the right person.
Leaderboard highlights (0–10): GPT-6 Astra tops at 8.97 ($0.693/task), followed by GPT-6.1 Sol at 8.65 and Claude Fable 5.2 at 8.17; Claude models lead on speed. The cheapest model, GPT-6 Luna at $0.007/task, scored 6.38 — beating Mistral Large 4, Gemini 4 Argon and last-place Grok 4.7 (4.51).
The authors argue proactivity is the next agent frontier but company data is too sensitive to benchmark publicly, so a synthetic company serves as a public baseline for teams choosing or building proactive agents.
More from coding & agent
- Omnigent 0.22.0 Adds Schedule and Event Triggers for Mixed Coding-Agent Software Factories — matei_zaharia · 2026-11-14
- Vertskim: Local MCP Server Gives Coding Agents a 3D Codebase Knowledge Graph — Queasy_Debate3423 · 2026-11-14
- Nursa's accountant and retired parole officer built real tools with AI — every · 2026-11-14
- We don't need more chatbots: an agent harness lead's field notes — threepointone · 2026-11-14
- Nexus launches an AI context engine that turns deployment knowledge into agent-queryable graphs — rohanpaul_ai · 2026-11-14
- Developer now runs 100% of his agents locally on a Mac Mini, up from 25% earlier this year — josh_bickett · 2026-11-14