New “expenditure horizon” metric compares human and agent cost efficiency on open-ended tasks
littmath · x · 2026-07-22
A proposed way to measure AI agent capability on continuously scored tasks: the expenditure horizon.
- It compares how human and agent performance improve as budget increases on open-ended optimization problems.
- The key point is the spend level where a human becomes more cost-effective than the agent.
- That crossover budget is defined as the agent’s expenditure horizon.
Related event: METR Proposes 'Expenditure Horizon' for AI Agent Evaluation(2 posts)→
More from Research
- Exponential launches RepoInject and BrowserInject benchmarks for prompt injection robustness — maksym_andr · 2026-10-27
- Odd eval result: Gemini 4.1 Flash gets more accurate when asked to control its CoT — dfrsrchtwts · 2026-10-27
- Redwood: CoT controllability evals are under-elicited; better prompts boost scores 2-3x — dfrsrchtwts · 2026-10-27
- Inverse-designed optical system solves MNIST using BeamZ — S_Conradi · 2026-10-27
- DeCert supervises generative reward models' rationales without reference explanations — Guiquan Sun · 2026-10-27
- VLM harnesses act as compositional generalizers, new blog post shows — a1zhang · 2026-10-27