JetBrains paper: coding-agent benchmarks should match real users' task flows
rohanpaul_ai · x · 2026-10-12
A new opinion paper from JetBrains Research argues that coding-agent benchmarks like SWE-Bench Pro diverge sharply from real usage: they hand agents one detailed issue and grade one final patch, with new features and bug fixes making up 93.8% of tasks.
Studying 4,782 real agent sessions in JetBrains IDEs (Production Sessions), the researchers found features and bug fixes were well under half of requests. Users mix questions about code, planning, review, refactoring, and execution, switching between task types mid-session—especially in longer sessions.
They built SWE-TaskFlow, which splits each benchmark issue into multiple turns and adds verifiable repository QA. In a pilot on 700 SWE-Bench Pro tasks, solving tasks sequentially roughly doubled agent cost with no stable change in resolve rate—showing the interaction protocol itself is an important evaluation dimension. Their advice: measure what your own users ask for, then shape the benchmark's conversation to match.
More from coding & agent
- Moli: open-source headless browser for AI agents renders pixels only on demand — abhishek__AI · 2026-10-12
- Open-source real-time object detection with YOLOv8, OpenCV, and threaded frame buffering — mariamamhan · 2026-10-12
- Claude Opus 5.5 makes a Blender short film about the last black hole — 3,000 lines of Python, no gen models — toni_btrain · 2026-10-12
- HF Engineer Runs pi-web on DGX Spark to Steer Local AI Agents From His Phone — lhoestq · 2026-10-12
- 10 prompting techniques that actually work, from few-shot examples to self-critique — thisdudelikesAI · 2026-10-12
- snowpaw: open-source self-hosted messenger where your contacts are AI agents — gnukeith · 2026-10-12