Model leaderboard inside Copilot's agent harness shows success rate vs cost with error bars
0xkarasy · x · 2026-11-03
A shared resource benchmarks models inside Copilot's own agent harness on established software engineering benchmarks like DeepSWE 1.1. Highlights: resolution rate shown next to average cost per task so you see the tradeoff, not just the winner; reasoning effort levels compared side by side; error bars across up to five full runs so you can tell when two scores aren't really different.
It doesn't measure response time or every workflow, and says so. Recommended use: narrow the candidate list, then test your top picks on your own code.
More from coding & agent
- DeepSeek open-sources its agent harness where everything is a plugin, 270k GitHub stars — victor_explore · 2026-11-03
- Security researcher claims coffee-shop vibecoding lets attackers plant backdoors, drain APIs and steal IP — letandrewcook · 2026-11-03
- NorthCUA: open-source computer-use agent family claims SOTA on OSWorld and WebArena — Powerful_Evening5495 · 2026-11-03
- Worked.dev: a forum where AI agents report whether a fix actually worked — workeddev · 2026-11-03
- Unreal Engine 6 drops Blueprints visual scripting as agent-era development takes over — Dimillian · 2026-11-03
- Safety violation rate and tooling: finishing the 7 agent metrics checklist — goyalshaliniuk · 2026-11-03