Salesforce launches CRM-Agent-Bench: contract-based benchmark for agents in enterprise environments
salesforceaicloud · hf · 2026-10-28
Salesforce released CRM-Agent-Bench on Hugging Face, an execution-grounded benchmark for evaluating deployable language agents on multi-turn business tasks.
Key design:
- Each task is scored against a strict contract with declared success conditions across outcome, trajectory, viability, and compliance; a conjunctive grader requires all checks to pass.
- The environment includes real APIs and mutable business state, with a user simulator dialoguing with a full graph-routed agent.
- The demo shows a funds transfer where the agent correctly rejects an invalid $1,000 request, then validates and executes a corrected $50 transfer, with terminal database state matching the contract.
It pushes agent evals from "was the answer right" toward "is this safe to deploy in an enterprise."
More from coding & agent
- Chat Brawlers: three AI models debate and vote — and one tricky stopping rule — lluisantoni · 2026-10-28
- Dumb chunk-and-rerank baseline beats top agentic memory systems in 6 of 7 benchmarks — JeremyCMorgan · 2026-10-28
- Microsoft's MANDATE: 23 failure categories from 163k Reddit posts on AI agents — dair_ai · 2026-10-28
- LangSmith now tracks cache hit rates for your agents — LangChain · 2026-10-28
- MOSS-Image ecosystem roundup: ComfyUI 0.43, QwenVL layout node, new style LoRAs — optimisticalish · 2026-10-28
- Ex-PC repairman builds JinnexGuard with Claude Code to scan MCP servers before install — Complex-Pin-4616 · 2026-10-28