Devin SWE-2 jumps to #3 on attention-to-detail coding test, tying Claude Sonnet 5.5
silasalberti · x · 2026-10-14
- Devin SWE-2 (max) jumped to #3 on EvalMapBench 6.7, a private coding benchmark measuring attention to unstated details, solving 25 of 43 maintenance tickets — tied with Claude Sonnet 5.5 (xhigh), behind only Claude Mythos 5.1 and Fable 5.1 (29).
- The benchmark writes tickets like a colleague would — feature only — with the rest hidden in CONTRIBUTING, docs, code patterns; a hidden verifier checks 284 detail checks met, 3 attempts × 20 min each through Harbor 0.24.
- Results are charted against price and output tokens to show cost-adjusted detail compliance.
More from coding & agent
- Agent traffic is a strange inference workload: long prefix caching beats raw tokens per second — rohanpaul_ai · 2026-10-14
- SWE-MISBEHAVIOR benchmarks coding agents' boundary violations, up to 98.5% of cases — NationalUniversityofSingapore · 2026-10-14
- KAIST's latent world model lifts GUI agent success rates up to 40.6% with 98.9% less latency — kaist-ai · 2026-10-14
- Opus 5.5 fuses playable games in a day; Musk says one-person game studios arrive next year — scottleibrand · 2026-10-14
- Replay your agent's real failures after every prompt, tool, or model change — strickvl · 2026-10-14
- Opus build a fully explorable three.js world in one 24-hour oneshot run — sidahuj · 2026-10-14