New Company Agent Bench compares Dots, Grok Bot, Claude Tag, Devin on company knowledge
philhchen · x · 2026-11-13
philhchen's team releases Company Agent Bench, a benchmark measuring how well company-assistant agents understand their company, comparing Dots, Grok Bot, Claude Tag, and Devin.
Key findings:
- Products scored 73–91% on authorship questions (ownership/expertise) but only 42–54% on questions about a person's interests and tendencies — understanding people's priorities, judgment, and working relationships remains the gap between a capable agent and an effective employee;
- API models with a simple harness outperform these packaged products.
Full write-up linked in the thread.
Related event: Company Agent Bench: Understanding People Is Enterprise Agents' Biggest Gap(4 posts)→
More from coding & agent
- Source, Sink, Drain: an IBM engineer's blueprint for running an AI software factory — TejasKumar_ · 2026-11-13
- Remote-driving someone's terminal agent with /voice tap, straight from your voice — alfcnz · 2026-11-13
- Expo launches cloud simulators built for coding agents, returning recordings automatically — EXM7777 · 2026-11-13
- Figma rolls out 3D transforms in open beta, with agent-assisted animation and MCP handoff — round · 2026-11-13
- The button is the API: Notion wires buttons to agents, and flips the pattern — bfrench · 2026-11-13
- How Salesforce HXL widgets render in Claude, ChatGPT, and Agentforce MCP clients — msrivastav13 · 2026-11-13