EdgeBench: Unveiling Scaling Laws of Learning from Real-World Environments
Deyao Zhu, Xin Zhou, Shengling Qin, Xuekai Zhu, Hangliang Ding, Shu Zhong, Zixin Wen, Zhonglin Xie, Chenhui Gou, Linxuan Ren, Yueyang Wang, Junfeng Zhong, Rui Liu, Tian Gao, Yangguang Lin, Jingyuan Zhang, Maojia Song, Xuan Qi, Jinhong Wu, Chenyang Zhang, Yinzhu Piao, Ziru Niu, Hongbin Lin, Lingxiang Meng, Peng Tang, Chengyao Tang, Shanyu Wu, Huanyu Zheng, Yu Liu, Liya Zhu, He Wang, Ming Ding, Ziyu Wan, Hao Liu, Sibo Wang, Haotian Zhu, Xintian Zhang, Nan Chai, Yipeng Liu, Panhao Lai, Sihang Yuan, Zixin Su, Ge Zhang, Wangchunshu Zhou, Yantao Du, Wenhao Huang, Guang Shi
cs.CL, cs.LG
2026-07-06
On 134 day-long tasks (~38k hours), aggregate agent scores fit a log-sigmoid at R²=0.998. Opus 4.8 hits 51.3 at 12h; learning speed rose ~8× in 221 days.
Pretraining scaling laws already tell you how loss falls as you add data and compute. Once an agent is deployed, it still has to poke a live environment, read feedback, and revise a plan. Whether that second kind of learning has a similarly clean curve was largely unmeasured.
Most benchmarks score an endpoint: did the answer land, did the patch pass, is the deliverable good enough. Suites such as MLE-bench and Agents' Last Exam already allow iteration, but the typical window is minutes to an hour or two, which is too short for exploration, strategy revision, and experience accumulation to show up. Measuring environment learning needs tasks that are long, richly instrumented, hard enough that current models cannot saturate them, and close to real work.
ByteDance Seed built EdgeBench: 134 executable tasks across six families (scientific problems and ML 39, systems and software engineering 36, combinatorial optimization 19, professional knowledge work 19, formal math 13, interactive games 8). Each run lasts at least 12 hours. Recorded human expert effort averages 57.2 hours and reaches 320. Tasks whose main difficulty is visual understanding or GUI operation are excluded, so perception is not confused with learning.
Feedback is dual-loop. The inner loop is local: tests, simulators, and validators inside a writable work container. The outer loop is a hidden judge in a separate container; only a submission returns calibrated scores or diagnostics. A host process also snapshots the workspace on a fixed schedule and scores it privately, so the learning curve can be drawn even between submissions.
They fit the cross-task average of best-so-far score with a three-parameter log-sigmoid, S(t) = Smax / (1 + (tmid / t)^β). tmid is the interaction time at half the fitted ceiling; β controls how sharply progress concentrates in log time. The accompanying theory treats a single task as frontier expansion on a latent score graph: unlocked score mass supplies reusable capability, locked mass is remaining opportunity, the expected cut is approximately their product, and a self-similar search volume puts the clock on a log scale. Individual tasks can plateau and jump; the smooth law is a population average.
Five frontier models are evaluated: Claude Opus 4.8, GPT-5.5, GPT-5.4, GLM-5.1, and DeepSeek-V4-Pro (preview), three independent 12-hour trials per task–model pair. GPT models run under Codex with a 256k compact window; GLM and DeepSeek under Claude Code at 200k; Opus mainly at 1M. Total interaction is about 38,000 hours. 51 tasks and the full evaluation harness are public.
On the 134-task average, every 12-hour curve hugs a log-sigmoid, with R² at least 0.997 and a mean of 0.998. The same form holds inside each of the six families. It remains precise on 28-hour runs (80 tasks, four models) and 72-hour runs (18 tasks, two models), R² at least 0.993. Fits on the first 6.5 hours forecast the rest of the 12-hour window at R² at least 0.997 and RMSE below 1.0 points. Among common S-curves, log-sigmoid has full-window RMSE 0.390; log-probit, log-Gompertz, and Weibull sit at 0.398–0.404; a log-linear baseline is 0.717.
12-hour leaderboard:
| Model | 2h | 12h |
| Claude Opus 4.8 | 39.0 | 51.3 |
| GPT-5.5 | 36.8 | 48.4 |
| GPT-5.4 | 29.7 | 39.3 |
| GLM-5.1 | 26.0 | 37.4 |
| DeepSeek-V4-Pro | 23.3 | 31.0 |
Opus leads every family mean. Code is the highest band (Opus 67.4, GPT-5.5 65.0); math has the widest gap (Opus 55.0, DeepSeek 14.1). More submissions do not automatically mean better learning: GPT-5.5 submitted the most (34.2k), GPT-5.4 had the highest effective-submission rate (22.2%), and Opus submitted less often than GPT-5.5 while finishing first.
Learning speed is measured on an 18-task slice where first-attempt scores are close (6.87±0.97). From GPT-5-Codex in September 2025 to GPT-5.5 in April 2026, two-hour gains rose about 8× over 221 days. A log-linear fit to the rolling top-2 frontier corresponds to a doubling roughly every three months.
Experience is worth points on its own. On 17 tasks, a continuous 12-hour Opus run that keeps the workspace and feedback history scores 43.0; the same budget split into six independent 2-hour restarts scores 36.1, a gap of 6.9. On 42 tasks, stretching Opus context from 200k to 1M lifts 12-hour score from 48.0 to 52.5 (+4.4); the 2-hour gap is already +5.8 and the curves stay roughly parallel. In a gravitational-wave reconstruction case study, GPT-5.5 made 224 submissions, only 27 of which raised the best-so-far by at least 0.1 points, taking the score from 42.8 to 67.0.
This is the first large, cross-domain measurement that treats post-deployment environment learning as a scalable object you can fit and extrapolate. If the log-sigmoid holds, early hours can forecast later scores, and models can be compared by tmid and β rather than a single cutoff.
The practical reads are blunt. A stateful continuous run beats equal-budget independent restarts. Workspace files and compaction already store state outside the prompt, and a longer context window still adds a few stable points. Flooding the judge does not fix sample efficiency; the fraction of useful updates, and whether those updates survive, matter more.
Treat it as a population law. Single-task curves are jagged, and fit error falls as more tasks are averaged. Useful for ranking and for studying how agents reuse experience. Too early to treat as a recipe for one agent in one repo.
The paper's own scope conditions are clear. Strong bottlenecks, widely dispersed task midpoints, heterogeneous frontier speeds across families, non-scale-free graphs, or a reachable score set that itself moves with longer interaction can all break the log-sigmoid. Fitted Smax is an effective ceiling inside the window, not an eternal cap.
A few weaker spots sit next to those caveats. Log-sigmoid only barely beats the other S-curves (0.390 vs 0.398), so the functional form is chosen on mechanism, not on a decisive fit gap. The three-month doubling is an 18-task matched slice over a short release window, not the full 134-task suite. Each setting has three seeds. Serving incidents are folded into the score, and GPT-5.4 availability dropped in the second half. The five models also sit on two harnesses and unequal context windows. Only 51 of 134 tasks are public, so the full-suite law cannot be independently re-run. The experience ablation uses 17 tasks and the context ablation 42; neither is the whole benchmark.