FinanceHarness: Autonomous Financial Deep Research Framework
Yijia Xiao, Rujun Han, Yanfei Chen, Zifeng Wang, Ke Jiang, Zhongying CuiZhu, Vishy Tirumalashetty, Wei Wang, Burak Gokturk, Tomas Pfister, Chen-Yu Lee
cs.CL, cs.AI, q-fin.CP
2026-07-30
FinanceGym is a point-in-time benchmark for financial deep research; the best agents reach 34% overall while the forward-looking part stays single digits.
General-purpose deep-research agents write rounded reports, but financial research is not that shape. An analyst does two very different jobs: weave together historical patterns, sector context, and numerical evidence, and forecast events that have not happened yet (next-quarter earnings, policy turns, deal flow). The first is retrieval plus synthesis; the second is extrapolation plus judgment, and general agents are nearly useless at it.
Evaluation is the other half of the problem. The cardinal sin in financial-forecasting benchmarks is information leakage: the test set contains material dated after the cutoff, so an agent looks like an oracle when it has simply peeked ahead. Most existing deep-research benchmarks do not guard against this.
The paper ships two things. First, FinanceGym, a benchmark of 400 expert-annotated thesis-driven questions with 2,464 rubric items. Each question has a cutoff date, and the linchpin is point-in-time (PIT) isolation: the FAISS corpus an agent can search is cut clean at the cutoff; nothing future is reachable. Scoring splits in two: pre-cutoff items answerable from before-cutoff evidence, and post-cutoff items that demand a forecast. A separate LLM judge scores each rubric item on a 0-4 scale. Experts validated the set at an 82% pass rate.
Second, FinanceHarness, an open-weight harness on a Qwen3.6-27B backbone. Its core design is a layered tool surface rather than a single generic search: the core loop (PIT search, source reading, citation composition, finalization) is always loaded; valuation, comparables, and risk tools load lazily so the prompt stays small. The runtime layer only dispatches and validates and holds no model intelligence, so the backbone is swappable. The harness is tightly coupled to the environment: tool calls hit the PIT sandbox's FAISS directly, and the rubric judge doubles as the training reward. That reward weights rubric scoring at 0.6 and an LLM judge over report coherence and trajectory quality at 0.4, trained with GRPO.
Seventeen baselines run on FinanceGym (normalized rubric mean, %):
| System | Overall | Pre-cutoff | Post-cutoff |
| Claude-Opus-4.7 | 34.1 | 50.1 | 9.9 |
| Gemini-3.1-Pro | 33.2 | 46.8 | 12.8 |
| FinanceHarness (27B open) | 32.4 | 45.7 | 11.8 |
| GPT-5.5 | 31.8 | 47.5 | 8.0 |
| Tongyi-DR (fine-tuned) | 28.2 | 39.7 | 10.7 |
Three readings. First, no system clears 40% overall, a wide gap below the expert 82% pass rate. Second and central: pre-cutoff sits at 40-50 while post-cutoff falls to 8-13 across the board, and that split is stable on every system. Since all agents use the same PIT corpus, the gap reflects the benchmark's core difficulty, not a retrieval misconfiguration. On known history these agents are respectable; on the future they are close to guessing.
Third is the value-for-money story. The fixed-backbone ablation on Qwen3.6-27B: bare model with search 25.3%, naive harness 29.6%, full harness 32.4%, harness plus GRPO 32.8%. The scaffolding alone lifts the same 27B model by 7.1 points (about +28% relative); RL training squeezes out only 0.4 more. A 27B open harness therefore beats every open-weight model, edges the best agentic-search system, and loses only to two closed giants. The authors also note that swapping the backbone within a fixed scaffold moves scores more than swapping the scaffold around a fixed backbone: financial deep research is still bottlenecked by the base model.
For agent and deep-research builders, two takeaways. One is the evaluation discipline: FinanceGym's point-in-time isolation is the template for any forecasting benchmark, because without cutting off the future the score is meaningless. The other is an engineering signal: scaffolding pays off far more than RL here, and a layered tool surface lets a mid-size open model approach closed flagships.
The bracing part is that pre/post split. Do not trust a deep-research agent to forecast; it is decent on history and close to a coin flip on the future.
The authors list two. The corpus covers only 2025 English-language web, so non-English sources, paywalled professional data, and structured filings are thin; specialized fine-tuned models are run on the authors' corpus-backed retriever rather than their trained-on web stack, so their scores are transfer results.
A deeper worry sits outside the paper's own list. The harness is tightly coupled to the evaluation environment (tools hit the sandbox FAISS, the rubric judge is also the reward), and the benchmark is the authors' own construction, so FinanceHarness playing on home turf is a structural advantage; transfer to real financial-research environments is not tested. The GRPO stage trains on 172 machine-curated samples and gains 0.4 points, which makes the RL component almost ornamental.