CurveShift: Is Agent Progress Scalar? Separating Level from Shape
Hanwen Xing, Pengyun Wang, BingXu Meng, Kumail Alhamoud, Xiang Li, Jicheng Wang, Xin Yu, Xinyang Han, Xiaomin Li, Philip Torr, Yuexing Hao
cs.CL, cs.LG
2026-08-01
Agent hard-task gains on METR are mostly a ceiling effect a scalar Rasch model reproduces; a +0.40-logit residual (18%→25%) survives only in scaffold-free competitive programming.
A popular claim in evaluation circles holds that newer models improve fastest on long, hard tasks, and that the capability frontier is migrating toward the hard end. METR's time-horizon data gets cited repeatedly to support claims that the duration of tasks agents can complete autonomously is doubling.
The question here is a measurement one. Is that "progress on hard tasks" an actual change in the shape of the difficulty-response curve, or an optical illusion produced by ceiling and floor effects as overall ability rises? A single scalar score that mixes overall level with curve shape cannot tell the two apart.
The authors borrow from psychometrics. Item response theory (IRT) models the probability a model solves an item as a function of its ability θ and the item's difficulty b; the simplest Rasch model differs by a single scalar θ−b, and adding a discrimination parameter gives 2PL. If only ability rises over time and the curve shape is unchanged, that is a level shift. If hard items gain more than level predicts, that is a shape change, statistically called differential item functioning (DIF).
The hard identification problem is the scaffold. On agentic benchmarks like METR and SWE-bench, new models are almost always evaluated with new scaffolds, so model era and scaffold era are collinear. A gain on a hard task cannot be assigned to the model or to its tooling.
LiveCodeBench breaks the confound. It runs competitive programming with no agentic scaffold, ships a human-labeled difficulty ordering, and its models are traceable by release date, giving the clean combination of dated models crossed with exogenous difficulty. The design is an anchored 2PL: freeze each model's ability estimate θ on easy and medium items, pin the hard-item discrimination α (joint estimation is degenerate), then estimate a hard-item era effect δ. The era breakpoint is September 2024 with o1-preview. δ is the residual gain that survives stripping out the ability, difficulty, and discrimination channels.
On METR, the apparent migration is a scalar artifact. In Table 2, the fastest post-o1 slopes sit in the 15–60 minute band (6→37 pp/year) and the over-4-hour band (0.3→21), which reads as hard tasks catching up. But a Rasch null that only lets a single ability dimension rise places its fastest-improving band in the same 15–60 minute range. The mechanism is mechanical: bands near the floor or ceiling compress their slopes, mid-difficulty bands sit on the steep part of the logistic, so uniform ability growth alone pushes the visual center of mass toward harder tasks. The only band the scalar null leaves a residual on is over four hours.
On LiveCodeBench a real hard-item effect survives. Sixty-six dated models × 1,055 problems (322 easy / 383 medium / 350 hard, base rates 0.83 / 0.45 / 0.18):
| Discrimination αhard | Hard-item era effect δ (logits) |
| 0.55 (binarized, deflated) | +1.36 |
| 0.83 (debiased) | +0.78 |
| 1.00 (equal discrimination, conservative headline) | +0.40 |
The conservative +0.40 logits has an item-cluster 95% CI of [+0.16, +0.67] and lifts the hard-problem solve rate from about 18% to 25%. It holds across the two contest ecosystems: Codeforces +0.79 [+0.40, +1.14], AtCoder +0.30 [+0.01, +0.59]. But the gain is led by the strongest reasoning models and lands on hard problems that need short reasoning, not tasks that need long-horizon autonomy.
The first takeaway is sober for practitioners. There is no clean evidence yet that agents are surging on long-horizon autonomous tasks. The much-cited METR migration is, to first order, a measurement artifact. A real hard-task gain exists, but it is bounded: competitive programming, short reasoning, on the order of +0.40 logits.
The second takeaway is methodological. Most agentic benchmarks upgrade model and scaffold together, so any hard-task gain can be booked to the model or to the harness. Progress read off SWE-bench-style benchmarks cannot be taken as pure model capability.
The authors keep the claim narrow. Clean identification rests on a single competitive-programming benchmark, so extrapolating to general agentic tasks comes with a discount. The binary post-o1 indicator bundles reasoning training, continued scaling, test-time compute, and agent-level test-time scaling, with no way to separate them. The size of δ also depends on the αhard assumption and ranges from +0.40 to +1.36; taking the most conservative value as the headline is honest, but it also signals real uncertainty in the effect size. IRT further assumes a unidimensional ability, plausible for competitive programming and less so for heterogeneous agentic tasks.