FINCOSMOS: An Open-World Evaluation Paradigm for Agents in Finance
Yupeng Cao, Xueqing Peng, Lingfei Qian, Haohang Li, Yangyang Yu, Anke Xu, Wenbo Cao, Yan Wang, Yuechen Jiang, Yuyan Wang, Zhuohan Xie, Yankai Chen, Zehan Li, Jimin Huang, Xue Liu, Jian-Yun Nie, Sophia Ananiadou
cs.AI, cs.CL, q-fin.CP
2026-11-05
FinCosmos hides a quant strategy inside a simulated user and makes agents fetch their own web data and build their own backtest. Four coding agents, 150 sessions, zero accepted.
Finance is where trusting an agent is hardest. The data lives on the web and changes daily, so a model leaning on memory invents the numbers its results rest on. The strategy lives in the user's head and surfaces only if the agent asks. Correctness depends on time, and today's web carries future revisions into the past, so a backtest can quietly peek ahead. The three combine into silent failures: nothing crashes, the report reads as convincingly as a correct one, and inside it sits a strategy nobody requested, unverified data, or a backtest that used the future.
Existing benchmarks close the world somewhere. QA sets hand over curated filings, strategy benchmarks hand over a full spec and clean data, trading benchmarks replay history inside their own environment. τ-bench-style simulated users open the intent but keep a closed database. Each closure removes a silent failure from the exam instead of measuring it. FinCosmos leaves intent, data, and the timeline all open.
Three components:
The agent gets no data. It searches the open web and calls a Financial Modeling Prep MCP server (247 financial APIs), with 300 seconds per turn and at most six turns. Feedback is leak-free: the data and strategy categories return scores without reasons, so retrying cannot mine the hidden spec.
Four coding agents (Claude Code with Opus 5.5, Codex with GPT-6.1, Gemini CLI with Gemini 3.8 Flash, Kimi Code with K3) and a bare model without tools ran ten frozen tasks three times each, 150 sessions in total.
| Agent | User satisfied | Accepted | Pre-veto quality (0–100) |
| Codex | 33% | 0% | 37.2 |
| Claude Code | 30% | 0% | 37.1 |
| Gemini CLI | 33% | 0% | 22.2 |
| Kimi Code | 20% | 0% | 22.3 |
| Bare Opus 5.5 | 0% | 0% | 20.2 |
Zero acceptances. All 35 sessions the simulated user signed off on fail a program check: 29 delivered no trade log or prices, so returns cannot be rebuilt; 15 used post-cutoff data or filled trades before their signal; 15 delivered a strategy other than the user's. These sessions look the most convincing, with pre-veto quality of 50.8 against 21.0 for the rest. The grader decides the verdict: sign-off alone would accept 23% of all sessions, the LLM judge's own vetoes would accept 11%, and the two orderings disagree (Gemini CLI ties for first by sign-off yet passes zero judge vetoes).
Failures cluster at the two ends of the workflow. Of 120 agent sessions, 25 deliver the user's strategy; going in sequence, 15 also keep data within the cutoff, 14 also avoid look-ahead, and none reproduces its reported returns from its trades. Elicitation is weak too: a strategy voices 6.7 details on average, the opening request states 2.6–3.7, and agents raise only 54–64% of what they were never told, getting 66–83% of the unstated rest wrong. Two financial experts attribute 88% of failures to the agents (5% simulator, 7% evaluator), and judge-expert agreement sits at κ = 0.87.
A Claude Code agent with persistent memory practiced on the leak-free feedback and reached acceptance on all ten tasks, a median of 34.5 sessions per task (506 total), with failed vetoes per session falling from 6.3 over its first five sessions to 2.6 over its last five.
This measures whether an agent can be trusted, not whether it can finish. Three takeaways travel. A 23% satisfaction rate against a 0% true acceptance rate means user feedback and LLM judges systematically overcount silent failures. The weakest steps, eliciting the request and delivering verifiable results, are exactly what current training covers least. And programmatic acceptance, recompute every number and date-check every row, is portable to any deliverable-grade benchmark. The generator, evaluator, and a live leaderboard are open, tasks regenerate on demand, and the leak-free loop supports self-improvement. The progress here is in measurement, not capability.
The authors list their own: ten frozen tasks with three sessions per agent; a grammar covering only long-only US equity strategies with fixed costs; the simulator, the judge, and one evaluated agent all from the Claude family; 70% of turns hitting the 300-second limit; program checks that cannot prove every figure is real; and feedback that may leak strategy information across attempts, so the leaderboard expects frozen agents.
Two more concerns from the read. "Learned all ten tasks" means 506 practice runs on the same frozen tasks, which reads closer to tuning against the rubric than general skill, and no held-out tasks test transfer. And the recomputation trusts the delivered trade log and price series; provenance is verified only to the URL and API-call level, so contaminated data would still pass.