Quant-Arena: Benchmarking LLM Agents for Quantitative Trading Research
Houde Qian, Zheng Jiang, Wenyi Li, Chenxi Xu, Xinjie Lin, Yuzhao Peng, Ruoyu Han, Youjie Zheng, Kaisen Yang, Qinhuai Na, You Fu, Calvin Xiao
cs.CL, cs.AI, cs.CE
2026-10-22
Nine frontier LLMs build A-share strategies from raw Level-2 data on Quant-Arena; on sealed 2024, 1 of 38 runs beats the 17.1% equal-weight market and the top scorer wins by holding cash.
Quantitative research is a loop: propose a market hypothesis, implement it as an executable signal, backtest it, decide what to keep. Existing benchmarks each grade one slice. AlphaQT-Bench scores trading-code correctness, AlphaBench scores formulaic factor generation, StockBench scores sequential trading decisions. No public benchmark answered whether a model can run the whole loop and produce a strategy that still earns on unseen data, and measuring it is genuinely hard: repeated backtests select for candidates that fit historical noise, so research-period scores and out-of-sample results routinely diverge.
Quant-Arena, from Navers Lab and Einsia.AI, makes the complete research run the unit of evaluation. A model builds its own features and stock-scoring code on raw A-share market data, iterates under bounded feedback, freezes one strategy, and the platform settles it on a sealed year. The authors call it the first public benchmark to grade the full loop on real market data.
The sealed year and the five-candidate cap exist to block multiple testing: no unlimited trial and error, no cherry-picking. Realized return dominates the weighting because strong signal statistics do not by themselves make a profitable portfolio.
Nine frontier models at every reasoning effort they support gives 38 iterative configurations. The gross equal-weight market returned 17.1% annualized in sealed 2024:
| Configuration | Sealed ARR | vs market | Total | Max drawdown |
| GPT-6 Astra (xhigh) | 10.7% | −6.3pp | 51.29 | −6.8% |
| GPT-6 Sol (medium) | 18.4% | +1.4pp | 49.80 | −35.9% |
| GPT-5.6 Terra (max) | 8.4% | −8.7pp | 47.58 | −28.8% |
| Kepler-1 (organizers' proprietary) | NAV 1.93 | — | 77.56 | — |
Thirteen of 38 configurations posted positive returns; only one, Sol at medium effort, beat the market, and it paid with a −35.9% drawdown. At each model's best-scoring effort, none beat the market. Four models ended above starting capital, led by Astra at 1.11; the other five closed between 0.88 and 0.97.
The diagnostics matter more than the ranking. First, RankIC barely tracks realized profit (Spearman 0.09 across 38 runs): GLM turned a RankIC of 0.046 into −7.8% annualized, while Opus at medium effort made 4.4% on a RankIC of −0.002. Decile returns explain it. For six models the second and third deciles earn the most, 24–33 bp per day, and returns sag in the top decile, exactly where the portfolio buys; Terra and Astra are the exceptions with monotonically rising profiles. Second, the champion, Astra at xhigh, wins on timing: it holds cash unless the median 5- and 20-session log returns of the universe are both positive, and sat out 75% of trading days in 2024. When the market bottomed at 0.704 on 6 February, its NAV still stood at exactly 1.000, a −1.7% drawdown against the market's −29.6%. The cost: it captured only 16% of the rebound (Terra captured 73%) and still trailed the market by 6.3pp for the year. That rule appeared only in the fifth candidate, after losses in both research years. Third, reasoning effort is not monotone: six of nine models peak below their highest effort, and the spread within a model reaches 14.49 points against 18.77 between models, so picking the effort setting matters nearly as much as picking the model.
Iteration splits the same way. Paired against one-shot runs, all nine models gain in total score (mean 36.47 to 43.55), but only four gain in sealed ARR, by a mean of 2.9pp; research-period scores rise 10.36 points against 5.67 for the sealed period. Part of what iteration learns is specific to 2022–2023, as the authors concede.
For agent evaluation, the protocol of bounded feedback, frozen submission, and sealed settlement is portable to other domains. For financial LLM work, the findings are blunt: RankIC does not predict portfolio return, so grading quant agents on signal metrics alone misleads, and the out-of-sample profitability bar remains uncrossed. Kepler-1's RankIC (0.085) is actually below Astra's (0.101) while its Sharpe is 2.98 against 1.89, which locates the gap in converting signal into positions.
The authors concede that part of iteration's gains are research-period-specific and that the sealed score's combination rule is unpublished. Beyond that: Kepler-1 is the organizers' own proprietary strategy, so the 77.56 ceiling cannot be audited; the sealed test covers one year in one market, and Astra's timing rule happens to fit 2024's crash-then-rally structure, with no evidence it survives another regime; singling out the one market-beating Sol configuration is itself a 38-way selection. The equal-weight reference is gross of costs, so the bar is higher than it looks, and market impact comes from a simulated order book rather than real execution.