Longtail-SWE benchmark ships 202 real-world SWE tasks across 13 underrepresented languages
_akhaliq · x · 2026-11-11
Nebius researchers released Longtail-SWE, a repository-level coding-agent benchmark covering 202 real-world software engineering tasks in 13 underrepresented languages — Clojure, Dart, Elixir, Erlang, F#, Haskell, Julia, Kotlin, Lua, OCaml, R, Scala, and Swift.
Key points:
- Each task derives from a real GitHub issue and matching PR, with an executable environment and tests
- A solver–judge–repair pipeline iteratively fixes tests and task descriptions that reject valid alternative implementations or accept wrong solutions
- Nine model configurations, including frontier models, were evaluated on task-solving performance, consistency across repeated attempts, and resource usage
- All instances, environments, construction/evaluation tooling, and agent trajectories are open-sourced, plus an easier task set for evaluating smaller models
Related event: Nebius Releases LongTail-SWE Benchmark for 13 Underrepresented Languages(3 posts)→
More from coding & agent
- Latent Space tests NaiveAI's 2,100 tok/s coding model: fast, verbose, promising — Latent Space · 2026-11-11
- Anaconda's LLMOps guide: retrieval quality degrades independently of the model — anacondainc · 2026-11-11
- Dev demos agent with 'durable object brain' plus on-demand computer, local or remote — samgoodwin89 · 2026-11-11
- Right-sizing intelligence per role: frontier reasoning at the control plane, adversarial verifiers at the gate — clmt · 2026-11-11
- MCP Apps ship with your server and run in any AI app supporting the extension — ClaudeDevs · 2026-11-11
- MCP Apps support lands in public preview as spec is standardized with community — ClaudeDevs · 2026-11-11