FINCOSMOS: open-world finance agent benchmark where none of 150 sessions passes
TheFinAI · hf · 2026-11-11
The Fin AI and collaborators introduce FINCOSMOS, an open-world evaluation paradigm for finance agents targeting failures existing benchmarks design away: data comes from the open web, the strategy lives in the user's head, and correctness depends on time — so agents fail silently with unverified data or backtests that peek at the future.
Design:
- A simulated user holds a hidden strategy sampled from a compositional grammar, revealed only when asked.
- The agent gets no data: it must acquire data from the web, build its own backtest under a data cutoff, and deliver verifiable results.
- Grading needs no reference answers: programs verify point-in-time correctness and recompute every reported metric; a run passes only if the user is satisfied and none of 15 veto checks fails.
Results: across 4 coding agents and one tool-less model over 150 sessions, zero sessions were accepted — every session the simulated user signed off failed a program check. Agents raised only 54–64% of the details they weren't told; human experts attributed 88.33% of failures to the agents. With FINCOSMOS feedback, an agent with persistent memory learned all ten tasks.
More from coding & agent
- Vibe42 update lets you mix terminals, remote desktops and browsers with sync across devices — yihui_indie · 2026-11-11
- Craft Anything: 27k-star open-source tool reverse-engineers and rebuilds any software end-to-end — mathemagie · 2026-11-11
- KAIST's WikiTrace Links Agent Memories to Evidence, Cutting Memory-Induced Degradation — kaist-ai · 2026-11-11
- n8n Labs open-sources Nod, a personal agent that survived all 216 prompt-injection attacks in WhisperBench — aakashgupta · 2026-11-11
- Researcher vibe-codes a tool to geolocate aircraft via doppler-shifted propeller noise — IgorCarron · 2026-11-11
- Sparrow's New Table Mode Extracts Skewed Scanned Reports Without Queries or Schemas — andrejusb · 2026-11-11