Repair, don't discard: Longtail-SWE triples task yield across 13 underrepresented languages

Longtail-SWE: High-Quality SWE Tasks for Underrepresented Languages

Vladimir Dobrovolskii, Ibragim Badertdinov, Alexander Golubev

cs.SE, cs.AI

2026-11-07

Longtail-SWE repairs flawed candidate tasks instead of filtering them out, growing from 68 to 202 tasks across 13 underrepresented languages; Opus 5.5 tops out at 81.5% Pass@1.

What problem this solves

Repository-level issue-resolution evaluation, where an agent gets a real repository plus an issue statement and must produce a patch that passes designated tests, is dominated by Python and a few mainstream languages. Multi-SWE-bench, SWE-PolyBench and SWE-Bench Pro widened the map, but Clojure, Erlang, F#, OCaml and Lua still have essentially no dedicated issue-resolution benchmarks. How coding agents cope with a different toolchain or programming paradigm has been guesswork.

The problem bites hardest where candidate pools are small. Tasks come from historical issues and merged PRs: the PR's own tests may lock in incidental details of the reference implementation and reject valid alternatives, descriptions may omit requirements no agent could infer, and thin tests let wrong solutions pass. The standard remedy is to filter suspicious candidates out, which leaves fewer tasks and biases survivors toward small code changes; the SWE-bench Multilingual release notes admit exactly that. Nebius argues the opposite: a defective task is usually repairable, and repairing beats discarding.

Method

Four stages (collection, construction, repair, finalization) built on SWE-rebench V2.

Handing blame assignment to the judge is what makes the loop work. Pass/fail alone cannot separate a defective task from a weak solver; the trajectory shows whether tests are rejecting a legitimate implementation or a wrong patch slipped through a coverage gap. Three hard constraints keep the task's intent intact: descriptions are append-only, the reference patch is untouchable, and test patches cannot carry product-code fixes.

Results

Nine configurations, three attempts per task, SWE-agent-style scaffold:

ModelPass@1Pass@3Mean tokens
Opus 5.5 (high)81.5289.114.48M
GPT 6.1 Sol (high)78.3882.181.75M
GPT 5.6 Sol (high)75.7487.132.84M
GPT 5.6 Luna (xhigh)71.4584.1613.01M
Kimi K3 (max)63.7076.2416.84M
DeepSeek V4 Flash (high)56.1171.7826.50M
GLM-5.3-Flash (max)48.1869.8038.28M
Qwen 3.8 27B (xhigh)42.7462.3823.06M
Qwen 3.6 27B (thinking)16.6723.767.20M

Opus 5.5 leads runner-up GPT 6.1 Sol by 3.14 points and still leaves 10.89% of tasks unsolved after three attempts. Efficiency spreads wider than scores: GPT 6.1 Sol averages 1.75M tokens and 27.4 steps, while GLM-5.3-Flash burns 38.28M tokens and 313.7 steps for a 48% Pass@1; Luna spends roughly three times Opus's tokens for a lower score. Qwen 3.8's pass@k over ten attempts climbs from 43.8% to 73.3%, leaving 26.7% never solved. Averaged equally across configurations, Erlang is easiest at 73.5% and Clojure hardest at 48.1%, a 1.5x spread; with 12 to 27 tasks per language, treat the ranking as indicative.

The repair effect itself, three comparisons:

ComparisonBeforeAfter
63 tasks, same Opus trajectories, tests only, Pass@129.1%67.2%
Same tasks, Qwen 3.8 pass@10 over ten attempts36.5%68.3%
End-to-end yield vs the SWE-rebench V2 path68 tasks202 tasks

Fixing tests alone, with trajectories and descriptions held fixed, lifts Opus by 38 points. Benchmarks that skip test repair systematically underreport. On yield: from 3,261 candidates the old funnel retains 68 tasks (2.1%) and the new one 202 (6.2%); re-annotated after repair, 196 tasks reach A-grade versus 66, a 2.97x factor. Kotlin's usable tasks go from 6 to 86, Scala from 2 to 78.

The judge got its own human audit: 65 tasks, two engineers, agreement of only Cohen's κ = 0.15, so even humans struggle to spot task defects. The three-judge protocol rejected 56 tasks and correctly caught 49 of 51 defective ones (precision 87.5%, recall 96.1%), at the cost of rejecting 7 of 14 acceptable ones. Separately, a single Luna judge reviewing the old pipeline's 364 built tasks flagged at least one quality issue in 90.1% and accepted only 9.9%, evidence of how many defective tasks install-only pipelines ship.

Why it matters

Benchmark builders can copy this recipe wholesale: wherever tasks are mined from historical PRs, mainstream languages included, 'trajectories as evidence, judge for blame, append-only repair' raises yield and quality together, and it is cheaper and more reproducible than expert annotation. Code is MIT, data CC BY 4.0, with trajectories, container images, and an easier companion set for smaller models all released.

For model selection it adds a measurement surface beyond the saturated Python leaderboards: the nine configurations span more than 60 points on identical tasks, languages add another 1.5x spread, and token efficiency diverges harder still. Keep difficulty in perspective: 81.5% Pass@1 means these tasks do not cap an Opus-class model, and calibration already removed tasks Qwen 3.8 solved three for three. This is the first reliable coverage of a blind spot, not a new difficulty ceiling for SWE evaluation.

Limitations

Acknowledged in the paper: judge specificity on acceptable tasks is 50%, so half the good tasks get unnecessary repair cycles and construction cost balloons; difficulty filtering ran on Qwen 3.8, whose reported scores may be inflated by selection; models that helped build the benchmark (the GPT 5.6 family, Kimi K3, Qwen 3.8, plus Opus 5.5 and GPT 6 models assisting post-evaluation review) also sit on the leaderboard, an effect the paper acknowledges but does not quantify, leaving GPT 6.1 Sol, DeepSeek V4 Flash, GLM-5.3-Flash and Qwen 3.6 as the only clean contestants; tasks come from public issues and PRs, so training contamination cannot be ruled out; and with 12 to 27 tasks per language, cross-language gaps reflect this sample and these configurations rather than inherent language difficulty.

Two further gaps: the 63-task pairing isolates only test repair, so the effect of appended description clarifications, and whether they quietly shift task intent, is not evaluated separately; and the judge's precision and recall come from a 65-task audit set, while the error rate of the final 202 tasks would need an independent audit the paper explicitly does not provide.

Terms

Source

Related papers

All paper explainers