ByteDance Introduces SWE-Bench ProMax for Multilingual Code Refactoring
ByteDance · hf · 2026-08-11
ByteDance introduces SWE-Bench ProMax, a new benchmark designed to evaluate AI coding agents on large-scale, multilingual code refactoring tasks.
- Background: Existing benchmarks like SWE-bench Verified are saturating rapidly. A recent audit found that nearly 60% of their unsolved instances contain flawed tests (e.g., rejecting correct solutions or checking unstated requirements), and frontier models can verbatim reproduce gold patches from training data.
- Features: It contains 170 real-commit instances across 7 languages (Python, Java, TypeScript, etc.). Every instance undergoes rigorous multi-stage curation, rewriting issue descriptions for precision and manually reviewing test suites.
- Results: Tasks average 11.4 modified files and 261.6 lines of code per instance. Frontier models tested under two agent scaffolds achieved a peak resolve rate of only 41.2%, confirming the benchmark remains a meaningful and unsaturated challenge.
Related event: ByteDance Introduces SWE-Bench ProMax for Multilingual Code Refactoring(2 posts)→
More from coding & agent
- Developer Uses Claude to Optimize ESLint Core Performance by 20% — DanielLockyer · 2026-08-11
- Abacus AI Releases Smaug-Agentic, Topping Open-Source Leaderboard for Agentic Coding — bindureddy · 2026-08-11
- Ouroboros: Self-Developing Coding Agent Tops Multiple Benchmarks — Anton Razzhigaev · 2026-08-11
- Evo-Bench: First Benchmark for LLMs' Ability to Autonomously Evolve Agent Harnesses — RUC-AIBOX · 2026-08-11
- Cloudflare Launches TypeScript-based CI/CD Pipelines, Ditching YAML — irvinebroque · 2026-08-11
- OpenGoat: Open-Source Framework for Hierarchical Multi-Agent Coordination — tom_doerr · 2026-08-11