ffmpeg-rebuilding benchmark will saturate in a fraction of the 3 years SWE-bench took
jyangballin · x · 2026-10-24
The author launched a site benchmarking models on rebuilding ffmpeg, showing numbers, trajectories and generated codebases. He predicts saturating this benchmark will take a fraction of the 3 years it took to saturate GitHub issue resolution benchmarks like SWE-bench.
More from Research
- Google's PLE benchmark: best AI agent scores only 50.6% on professional engineering deliverables — dair_ai · 2026-10-24
- Mathematician Danny Calegari: LLMs Mirror You Back, But Behave Better — fortnow · 2026-10-24
- No One Is Impressed by Good Results: Why AI-Proved Theorems Teach You Nothing — fortnow · 2026-10-24
- If Claude Had the Idea, Say So 'For Your Own Personal Soul': AI and Math Ethics — fortnow · 2026-10-24
- Top Mathematician Shows How He Uses Claude to Do Research and Build Talks — fortnow · 2026-10-24
- Quant hallucination forensics: fabricated prompt injections, pirate puzzles and a phantom Windows attack — Consumerbot37427 · 2026-10-24