OfirPress: SWE benchmarks are all full-auto, but we need human-in-the-loop evals
OfirPress · x · 2026-11-06
Princeton researcher Ofir Press argues that nearly all current SWE benchmarks target full-auto settings, while much real-world work is human+AI collaboration — so the field needs human-in-the-loop benchmarks. He also defended his mini benchmark (130 lines) against benchmaxxing concerns, noting the hard part is figuring out how to design such evals correctly.
Related event: SWE-bench Author Responds to Benchmark Gaming Concerns(2 posts)→
More from coding & agent
- FastAPI creator tiangolo says he now uses voice-driven Codex way more after jxnlco's talk — jxnlco · 2026-11-06
- AWS WAF + Stripe lets sites charge AI agents via HTTP 402, with fiat payouts — jeff_weinstein · 2026-11-06
- The computable enterprise: capturing business logic that lives in employees' heads — vishalmisra · 2026-11-06
- Claude skills are useless — until you hook them to your actual business — aryanXmahajan · 2026-11-06
- Do you use MCP firewalls with Claude, Codex or Cursor in production? — vdobroskok · 2026-11-06
- Firecrawl launches Alexandria agent data layer with Benzinga, announces $75M Series B — devdigest · 2026-11-06