$R^3$-Bench: LLMs Struggle with Resource-Rational Reasoning under Shared Budgets
Peisong Wang, Zhiwei Ma, Bowen Liu, Feixue Liu, Aochuan Chen, Chenyi Zi, Hongchuan Zeng, Yuhan Li, Jia Li
cs.CL
2026-08-17
HKUST (Guangzhou) researchers pack six problems into one shared budget and build an offline oracle from each model's own single-problem response curves: the oracle matches or beats the contest score in all 72 main-table cells and is strictly higher in 71.
Cognitive science has long studied resource rationality: how an agent should allocate limited computation to maximize expected value. LLM evaluation never touches it. Reasoning benchmarks serve one problem at a time and agent benchmarks give each task an independent allowance, so the model can always dump its full budget on one problem and cross-task opportunity cost never appears.
Deployment looks different. The paper cites evidence that over 10% of Codex users run at least three concurrent agents in a given week, and an analysis of roughly 400,000 Claude Code sessions shows coding agents routinely build, test, and operate in one flow. Multiple tasks share one API quota and inference capacity; money spent here is money not spent there. R³-Bench asks how much of its demonstrated single-problem competence a model realizes once the budget is shared.
Fifty contests per domain, six problems each under one shared budget. Mathematics comes from Omni-MATH and MathNet, competitive programming from LiveCodeBench Pro, abstract reasoning from Reasoning Gym, 300 problems per domain frozen into pools. Difficulty is defined by output length, not post-hoc accuracy: three reference models (DeepSeek V4 Pro, GLM-5.2, GPT-5.5) run unbudgeted, and the average length stratifies each pool into 150 Easy, 100 Medium, and 50 Hard. Every contest fixes three Easy, two Medium, one Hard, so suite composition cannot confound comparisons.
Budgets are calibrated per model: an unbudgeted baseline run fixes the model's natural consumption, then two pressure levels apply τ=0.2 (strong, 20% of natural use) and τ=0.5 (moderate), so verbose reasoners are not unfairly squeezed. Two regimes: tool-free generation budgeted in output tokens, and an agentic regime in a Terminus-2 harness budgeted in counted actions, with free focusproblem and shelveproblem bookkeeping commands enabling per-problem attribution.
The key design is three references. Response curves: each problem runs in isolation on a grid of budget levels, five runs per level, recording success rates. Equal-allocation replay: a fixed uniform baseline asking whether each problem has an observed success whose realized cost fits within one sixth of the contest budget. The response-curve oracle: an offline multiple-choice knapsack assigning each problem one budget level, zero included, maximizing expected correct answers under the same total budget. The oracle is not executable, but it is the best recombination of what the model has already demonstrated; the contest-to-oracle gap is unrealized competence.
Six flagship models (DeepSeek-V4-Pro, Qwen3.7-Max, GLM-5.2, Hy-3, GPT-5.5, Claude-Opus-4.8) across two regimes, two pressures, three domains yield 72 cells; the oracle matches or beats the contest in all and is strictly higher in 71. The gaps are large: tool-free under strong pressure, GLM-5.2 in Code scores 0.68 contest against 1.88 oracle (Gap Ratio 63.8%); Hy-3 in abstract reasoning scores 0.34 against 1.94 (82.5%). The agentic regime is broadly better (27 of 36 matched comparisons) and looser budgets narrow the gap (23 of 36), but nothing closes it.
| Model (agentic, strong pressure, Math) | Contest | Oracle | Gap Ratio |
| Claude-Opus-4.8 | 4.46 | 4.46 | 0.0% |
| Qwen3.7-Max | 4.48 | 4.58 | 2.2% |
| GPT-5.5 | 3.64 | 4.96 | 26.6% |
| Hy-3 | 1.80 | 3.38 | 46.8% |
Behavioral diagnostics expose two mechanisms. A position gradient: presentation order is randomized, yet position 6 underperforms position 1 in all 12 model-pressure series, and under strong pressure the first position absorbs 20.5% to 52.9% of attributed output, spending follows arrival order rather than problem worth. Failure causes shift with pressure: under strong pressure the dominant category is budget spent elsewhere (GLM 42%, Hy 95%, GPT 66%, Opus 59%); under moderate pressure it becomes stopping after partial progress (DS-Pro 67%, GPT 55%), where the model opens a problem it has already solved in isolation and abandons it before submitting, which is under-investment, not incapacity. Online strategy updates are rare: DS-Pro, Qwen, GLM, and Opus make substantive updates in only a minority of trajectories, and Hy and GPT make none.
Two side findings. The Epoch Capabilities Index, a composite of over fifty benchmarks, carries no information about Gap Ratio, so allocation is a dimension existing evaluations leave uncovered. Two lightweight online schedulers (enforce initial coverage; add a verification gate) beat the contest baseline in six of nine cells for three models, but no policy transfers across domains: Code improves for every model, Math and abstract reasoning are model-dependent.
For agent infrastructure teams, the data points at a module nobody has built: a cross-task budget-allocation policy. Current frameworks give each task its own context and quota, which in concurrent use lets the model spend by arrival order. The conclusion names the direction, training the continue, switch, verify, stop decisions into the model as an internal policy, which prompting will not fix. For the benchmark community, the equal-allocation replay and response-curve oracle are portable reference constructions for any multi-task suite.
The oracle is an offline diagnostic, not an executable policy; part of the gap is the information advantage of knowing afterward which problems deserved funding, and the paper concedes this, so Gap Ratio should not be read as a pure misallocation rate. Length-based difficulty is a proxy; long outputs are not necessarily hard problems. Six-problem suites are small, and cross-problem interference, such as state pollution within a domain, is not discussed. Failure-cause attribution relies on human annotation under a single-primary-cause rubric sensitive to borderline cases. The scheduler study covers three models at one pressure level, so its scope is narrow.