50-task Salesforce agent eval: GPT-6.1 Sol scores 72%, well behind Claude Opus at 87%
TruthDeep8659 · reddit · 2026-10-28
A developer ran five models through 50 tasks in a Salesforce dev org, grading on actual database changes rather than model replies.
- Overall pass rates: Claude Opus 87%, Sonnet 85%, DeepSeek 82%, Haiku 80%; GPT-6.1 Sol trails at 72% — the only model clearly behind
- GPT-6.1 strengths: caught every probability reset after stage changes (which DeepSeek and Haiku mostly missed), best at reporting automatic Salesforce changes in long jobs (21 of 36)
- Failures clustered in specific tasks, like undoing a "won" deal without restoring its original close date
- On L5 "broken" scenarios all models drop sharply (Opus 48%, GPT-6.1 35%)
Full write-up plus open-source code and transcripts (crm-agent-bench) are available.
Related event: Claude Opus tops Salesforce agent benchmark at 87%(3 posts)→
More from coding & agent
- AI scientist who built 400+ agents maps 15 essential memory patterns, with open-source links — MaryamMiradi · 2026-10-28
- Agents built a custom inference engine: AMD MI350X serves GLM 5.3 FP8 37% faster than NVIDIA B200 — yisongyue · 2026-10-28
- Kiro explains how 3 engineers and agents keep 1,000 PRs a week stable with 80+ automated checks — rseroter · 2026-10-28
- Single vs multi-agent: same task, 4 setups, 3x cost gap measured — latent_lurker8587 · 2026-10-28
- Mapping the stack that gives agents inboxes, phone numbers and wallets — Similar_Confusion534 · 2026-10-28
- Solo dev's plan to work around Meta's WhatsApp error 3441038 with a fresh number — Fit-Stretch-182 · 2026-10-28