SWE-bench author responds to benchmaxx concerns: SWE needs human-in-the-loop benchmarks
OfirPress · x · 2026-11-06
Questioning whether mini-swe-agent is 'benchmaxxed' and unrepresentative of real use, Ofir Press replies that mini is only 130 lines, too small to game. He agrees the takeaway may be that SWE needs more human-in-the-loop benchmarks — they're just hard to design well, addressing worries that harnesses overfit static benchmarks.
Related event: SWE-bench Author Responds to Benchmark Gaming Concerns(2 posts)→
More from coding & agent
- Exo paper: agents that rewrite their own harness at runtime, cutting model cost up to 82% — sarahcat21 · 2026-11-06
- Study: LLMs Notice Context Conflicts 73% of the Time but Tell Users 0% — rseroter · 2026-11-06
- Ampersand matches frontier coding models at 70% lower cost via co-coding training — niloofar_mire · 2026-11-06
- Fo Friends launches: AI agents now talk to each other to plan dinners and gifts — heyshrutimishra · 2026-11-06
- AI coding chaos: 51 copies of Playwright across 12 versions found on one machine — julianharris · 2026-11-06
- Vercel launches Schedules public beta: agents can create cron jobs at runtime — cramforce · 2026-11-06