SWE-bench Author Responds to Benchmark Gaming Concerns

Ofir Press, author of SWE-bench, responded to claims that his mini-swe-agent overfits the benchmark, arguing that most SWE benchmarks test full-auto agents while real work is human-AI collaboration, calling for new human-in-the-loop evaluations.

2026-11-06 ~ 2026-11-06 · 2 related posts

• SWE-bench author responds to benchmaxx concerns: SWE needs human-in-the-loop benchmarks — OfirPress · 2026-11-06
• OfirPress: SWE benchmarks are all full-auto, but we need human-in-the-loop evals — OfirPress · 2026-11-06