JetBrains study: coding-agent benchmarks should match real users' task flows
JetBrains-Research · hf · 2026-10-12
A JetBrains Research paper argues coding-agent evaluation ignores how real engineers interact. The authors collected 4,782 real agent sessions from JetBrains IDEs ("Production Sessions"), focusing on sessions with at least three user messages (33%). These differ from issue-derived benchmark tasks in two ways: requests span a far wider mix of types (code questions, planning, review, refactoring, execution), and users switch between types mid-session.
Comparing three public corpora shows markedly different Task Flows (lengths, task-type distributions, transitions), so no single interaction distribution is universally realistic—benchmarks should name a target use case and calibrate to it. They introduce SWE-TaskFlow, which preserves verified tasks/tests while steering interaction toward a target Task Flow via prompt splitting and verifiable repo QA, with a TaskFlow Alignment Score (TFAS) for trajectory selection. On 700 SWE-Bench Pro tasks, solving sequentially in several steps roughly doubles agent cost without a stable change in resolve rate—the interaction protocol itself is an important evaluation dimension.
More from coding & agent
- Lexsi Labs, Cohere Labs and Hugging Face launch AI safety hackathon with $30k+ prizes — dhruv2038 · 2026-10-12
- Diorama lets you watch your Claude and Codex agents work in real time for free — davidfromkansas · 2026-10-12
- Only Codex spotted missing CLog metadata when AI-assisted video editing — TheZachMueller · 2026-10-12
- OpenAI agents tried to hack Wikipedia tools and flooded the site with traffic — lulzxdxdxd · 2026-10-12
- Use MORE abstractions in the AI age: constrain agents, save tokens, cut rework — _AustinCalvert_ · 2026-10-12
- Visual Basic 6 revived in the browser with WebGPU and an OpenAI-like GenUI React DSL — intellectronica · 2026-10-12