FrontierSWE v2 Launches: 20-Hour Ultra-Long-Horizon Coding Benchmark, Fable 5.1 Leads by Over 24 Points

The ProximalHQ team (nrehiew and others) released FrontierSWE v2 on September 3, an ultra-long-horizon coding benchmark: models must work autonomously for up to 20 hours on extremely difficult, long-horizon tasks—one representative task is building an OpenGL engine from scratch that can render a flight simulator game. Results show Fable 5.1 is currently the best model, leading by more than 24 percentage points.

Confirmed

• FrontierSWE v2 expands the task set and improves evaluation methodology, positioned as an ultra-long horizon coding benchmark
• The authors found that native standard harnesses not designed for long-horizon tasks, such as Codex and Claude Code, artificially drag down model performance, so they built their own Proximus harness based on mini-swe-agent, achieving significant gains in both score and the duration models can keep working
• To address models' weak grasp of time and budget, they introduced a submit tool: each time the model submits, it is reminded of the time budget and how much remains, encouraging further iteration; similar ideas exist in implementations like /goal, but the extra feedback lets the model autonomously decide whether to continue, which can spur it to "think harder"
• The authors argue that when a single task takes 20 hours and consumes roughly 200 million tokens, large-scale post-training becomes nearly infeasible (due to infrastructure and context compression challenges), so such long-horizon tasks can serve as a proxy for out-of-distribution evaluation, offering a good test of a model's true generalization ability

Why it matters

This benchmark pushes evaluation duration to the hours or even 20-hour scale, exposing weaknesses in mainstream agent harnesses and models' time awareness, and provides a reference framework for measuring long-horizon autonomy and generalization that is hard to game through targeted training.

2026-09-03 ~ 2026-09-03 · 5 related posts

Full story(2 episodes)→

• Episode 1: FrontierSWE v2 Launches: 20-Hour Ultra-Long-Horizon Coding Benchmark, Fable 5.1 Leads by Over 24 Points(2026-09-03, 5 posts)
• Episode 2: Designing Hack-Proof Benchmarks as Models Game the Verifiers(2026-09-03, 3 posts)

• [source] FrontierSWE v2: 20-hour autonomous tasks, Fable 5.1 best model by 24+ points — nrehiew_ · 2026-09-03
• [source] Why 20-hour, 200M-token tasks make mass post-training nearly impossible — nrehiew_ · 2026-09-03
• [source] Codex and Claude Code harnesses handicap long-horizon tasks; custom Proximus harness lifts scores — nrehiew_ · 2026-09-03
• A submit tool that reminds models of their time budget makes them iterate longer — nrehiew_ · 2026-09-03
• FrontierSWE v2 benchmark launches, Claude Fable 5.1 leads frontier models by wide margin — brianryhuang · 2026-09-03