LongTail-SWE benchmark: 202 niche-language coding tasks, GPT 6.1 Sol loses to 5.6 Sol
teortaxesTex · x · 2026-11-11
The newly released LongTail-SWE benchmark tests models on 202 real tasks across 13 rarely-evaluated programming languages (Elixir, OCaml, Haskell, Clojure, etc.), built from real issues/PRs with descriptions and tests rewritten via agent rollouts for fairness. All tasks ship on Harbor with Docker images; trajectories are on Hugging Face.
Key findings:
- The closed-vs-open gap is widest on functional languages: Opus 5.5 and GPT 6.1 Sol show no drop, while DeepSeek, GLM and Qwen fall further behind (Overall: Opus 5.5 81.5%, GPT 6.1 Sol 78.4%, DeepSeek 56.1%, Qwen 3.6 16.7%).
- teortaxesTex flags anomalies: GPT 6.1 Sol loses to the older 5.6 Sol on several languages, sometimes even to 5.6 Luna (e.g. Dart 61.9 vs 81.0, F62.5 vs 81.3), suggesting inconsistent effort settings; Lua is likely saturated at 90%.
- Language-level wins are scattered: both leaders hit 100% on Erlang, Opus leads Clojure (82.1), and Kimi K3 tops R (81.0).
Related event: Nebius Releases Longtail-SWE Benchmark for 13 Underrepresented Languages(4 posts)→
More from Models
- MMDM Benchmark: 5,448 Questions Show Decision Models Cap Out at 36.8% on Hard Multi-select — CUHKDBGROUP · 2026-11-11
- SimpleBench trend breaks: newest models beat trajectory on common sense — ChrisGPT · 2026-11-11
- Fable 5.2 vibe-coding test: two prompts yield a photoreal FPS and a 3D Battle Breakout — vista8 · 2026-11-11
- $250/month now buys you team-of-engineers-level intelligence on tap — ethanniser · 2026-11-11
- Opus 5.5 dominates ValsAI's new Watershed cyber eval, breaking through at ~$17 — teortaxesTex · 2026-11-11
- Jon Finger: binary-operating coding models could be a video tokenization strategy — mrjonfinger · 2026-11-11