Nebius releases Longtail-SWE: 202 repo-level SWE tasks across 13 underrepresented languages
nebius · hf · 2026-11-10
Nebius released Longtail-SWE, a benchmark of 202 issue-resolution tasks across 13 languages poorly covered by existing repo-level benchmarks: Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift.
- Built with a solver–judge–repairer pipeline: a solver attempts each task, a judge reviews the task plus the solver's trajectory and test results, guiding revisions that fix tests rejecting valid alternative implementations or accepting wrong solutions — while preserving difficulty and intent.
- Evaluated 9 model configurations including frontier models on task-solving performance, consistency across attempts, and resource usage.
- Fully open: benchmark instances, executable environments, construction/eval tooling, and agent trajectories, plus an easier task set for smaller models.
More from coding & agent
- Re-running an old castle prompt: Fable 5.2's procedural Blender output is a visible jump — aitrendz_xyz · 2026-11-10
- Fable 5.2 plus a few Opus subagents produces a complex 3D scene in ~30 minutes — aitrendz_xyz · 2026-11-10
- Leetcode interviews will die as agentic coding rounds become the new hiring standard — rachittshah · 2026-11-10
- 10 open-source GitHub repos for voice AI: from local ASR to zero-shot voice cloning — Shruti_0810 · 2026-11-10
- As Arc sunsets, a dev builds Earful, an AI tool that aggregates product reviews into verdicts — BLCNYY · 2026-11-10
- Claude's Chrome Extension Slammed for Being Unable to Control Already-Open Tabs — Angaisb_ · 2026-11-10