FM-Bench: A Benchmark for Long-Horizon Management with Competing Agents
Tianyou Wang, Chongyang Gao, Kezhen Chen, Chen Dong, Yinghao He, Donghan Li, Wangcheng Xu, Hongjiu Zhang, Chi Li
cs.AI
2026-08-19
Fifteen frontier models run a football club for 20 in-game years. Claude-fable-5 leads solo at 90.94 and the Arena at 76.26; tokens do not rank, and none learn hidden prices.
Most agent benchmarks are short-horizon: one GitHub issue, a handful of tool calls, one web deliverable. The environment barely pushes back, errors rarely compound across years, and no rival is bidding for the same asset. Business sims stretch time, still against scripts or replayed markets. Management is a different job. Information stays hidden, payoffs arrive seasons later, rivals mark up whatever strategy you reveal, and a board scores results and financial discipline together.
FM-Bench instantiates that job in a deterministic football-manager engine. An agent runs one of 16 clubs for 20 in-game years, roughly 340 to 400 decision stops, 26 tools. Draft budgets are equal. True player ability stays behind biased scout bands. The transfer market raises hidden asks after rejections and marks up repeat counterparties. The board jointly scores results and finances. Firing or insolvency does not end measurement: at most three discounted revivals. Every stop is a fresh conversation. The only state the agent carries is a notebook it writes itself, so memory curation is part of the test. The engine accumulates honors, value added, and squad value. No LLM judge, no human rater.
Two tracks share the engine. Solo: one model against 15 tiered scripted clubs. Arena: 15 models plus a scripted anchor in one world, with sealed bids, equal endowments, and capped revival so a single shared run is a valid ranking. Anchors include an oracle that reads hidden state, a disciplined blind heuristic, idle, and random. Six first-play humans ran the same solo track.
All 15 models finish every 20-year seed. Blind scripts mostly die: heuristic 17.05, idle -0.90, random -17.21. Solo oracle 95.54±4.68. Claude-fable-5 leads at 90.94±5.20 on 24M tokens, about 95% of the oracle. Open-weight kimi-k2.6 is second at 88.49±0.15 with almost no seed spread. Gpt-5.6-terra at 86.66 beats gpt-5.6-sol at 86.40. Claude-haiku-4.5 sits at 36.90±22.73 and can swing more than 20 points across seeds. Scale, list price, and vendor do not order this board. Token spend correlates with score at Spearman -0.19. Year-5 rank correlates 0.19 with the finish; deepseek-v4-pro leads through year 10 and ends 12th. A shorter horizon would have ranked a different field.
Three behaviors track the score on every seed: cutting slow-payoff academy and facility spend near the end (rs=-0.58), not letting cash sit idle (rs=-0.50), and opening contract renewals early (rs=+0.45). Price discovery fails across the board. The oracle closes every buy on the first offer. The field median is 30 offers per completed signing, Fable needs 9, gemini-3.5-flash needs 73. Notebooks fail in two opposite modes: gpt-5.6-sol at cosine 0.91 is an append-only archive; claude-sonnet-5 at 0.20 rewrites the plan each season. Fable sits at 0.39. Four of six humans die out; the best finisher scores 74.64, the floor of the model board.
In the Arena Fable still leads at 76.26, muse-spark-1.1 second at 62.47. The league title rotates among ten models; the champion defends in 2 of 19 season transitions. The scripted anchor exits in year 3. Fable's bid rate rises from 0.5 offers per season in solo to 4.6 in the Arena. Claude-opus-4.8 writes that idle cash should become young players and still finishes holding about 2,100M in reserves. Knowing the policy and executing it are different skills.
This is one of the few agent evals that binds a long horizon to adaptive rivals under a mechanism score. Bigger, pricier, or more talkative models do not automatically manage better. For anyone shipping a long-running agent, the portable signals are concrete: endgame discounting, cash deployment, early renewals, and whether the notebook is an archive or a seasonal rewrite. Memory curation and price learning are unsolved across the whole roster, more informative than another short-task point.
The Arena is one seed in the main text; how stable the seating is to world noise is unknown. The oracle is a privileged script, not an optimum, so 95% of it is not a ceiling. The roster is a July 2026 flagship snapshot with internal names that are hard to reproduce across papers. Football rules and scout bias are author-calibrated; a different industry could reshuffle the order. The six humans are first-play, not expert-versus-expert. The engine strips real names to block memorization, and with them any brand premium of a real transfer market.