OpenAI's $0.11 model beats the priciest LLMs in new Slack agent benchmark
heyshrutimishra · x · 2026-11-13
Deepnote introduced the Agent Initiative Benchmark: they built and open-sourced PageMantle, a synthetic company with 6 years of detailed history (financial model, CRM records, past-due invoices, Linear tickets, and real events like the SVB collapse). 12 frontier models were tested as proactive agents listening in on the company's Slack across 50 workplace situations.
Key result: OpenAI's cheapest GPT-6 Luna ($0.007/task) beat the three most expensive models; priciest Claude Fable 5.1 costs $3.02/task — a 430x price gap with only a 1.1-point score difference on a 10-point scale. Caveat from the poster: 50 tasks is still a small sample before picking a model off it.
More from Models
- Chinese Paper Maps CoT's Hidden Topology: Reasoning Behaves Like a Molecule, Not a Line — mikeflache · 2026-11-14
- 1.26M blind votes show users judge AI more harshly each month — models aren't declining — rohanpaul_ai · 2026-11-14
- Meta's unannounced Muse-Spark-1.4 model spotted in GitHub PR with real API usage logs — testingcatalog · 2026-11-14
- MathArena First Non-OpenAI #1: Gemini 4 Carbon Overtakes GPT-6.1 Sol — vedantmisra · 2026-11-14
- DeepMind unveils AlphaProof Code: AI-assisted formal verification for real-world software — Dense_Primary629 · 2026-11-14
- Dev trains 46M-param native Luganda model Pearl Chat, still collapses after 2 turns — weskambale · 2026-11-13