Forecasting benchmark simulates luck ceiling; LLM evals need the same

Crescitaly · reddit · 2026-08-26

Drawing from FINAL-Bench's financial forecasting challenge, this critique highlights flaws in current LLM Agent evaluations and proposes improvements:

FINAL-Bench's Innovations:

Scores future outcomes instead of backtests.
Preserves all entrant records, not just winners.
Simulates 20,000 random players to publish the 95th-percentile "luck ceiling".
Self-tests the scorer (including no-lookahead tests) for fairness.

Recommendations for LLM Agent Evaluations:

Establish random or simple-policy baselines.
Generate score distributions via repeated seeds.
Pre-publish a chance ceiling for performance.
Use a frozen scorer with adversarial self-tests.
Retain all failed runs.
Include cost and latency in rankings.

A single leaderboard point cannot distinguish between a skilled model and one that simply got a lucky trajectory.

Original post →

Forecasting benchmark simulates luck ceiling; LLM evals need the same

More from coding & agent