Which benchmarks still stump frontier models? SWE-sweep tops out at 9.5%
Background-Wafer-548 · reddit · 2026-11-11
A Reddit thread collects benchmarks where frontier models barely make progress. Most reported scores now sit above 20-40%, but SWE-sweep — a hint-free bug discovery and remediation benchmark — tops out at just 9.5% (Astra), with Fable 5.1 at 2.9%. The discussion highlights which tasks remain near-unsolved as standard leaderboards saturate.
More from Research
- Prism capability extraction: specialists carved from Qwen 27B at <20% memory match full model — tokenbender · 2026-11-11
- Data Colada Flags Fraud in Fifth Francesca Gino Paper, No Forensics Needed — danielrock · 2026-11-11
- Fable 5.2 tops GDPx chip-design benchmark, but task authors see zero gains — ycombinator · 2026-11-11
- Fortnow asks Claude about complexity in the Aronson-Ambainis proof: Promise-BQP-complete — fortnow · 2026-11-11
- Backdooring AI agents by reprogramming fewer than 10 dormant neurons — no training needed — DimitrisPapail · 2026-11-11
- Duke and Princeton paper: 10 small parallel agent teams beat best-of-N and one giant team — rohanpaul_ai · 2026-11-11