Backtesting an LLM Is Hard: virattt Hides Tickers and Dates to Curb Data Leakage
virattt · x · 2026-09-28
virattt, author of the open-source AI Hedge Fund project, points out a fundamental problem with backtesting LLM-driven trading strategies: the returns may already be baked into the model's weights, since training data contains historical market action.
To limit this leakage, the project's backtester now hides the ticker, dates, and position size from the model, making it harder for the LLM to recall answers from memory rather than reason.
More from coding & agent
- Baseten's Guide: Evaluate Open Models for Coding Agents on Your Own PRs — baseten · 2026-11-16
- Google Research open-sources PaperOrchestra, a multi-agent framework for writing AI papers — google-research · 2026-11-16
- ffmpeg-skill gives coding agents local video editing via 42 FFmpeg tools and MCP — kajisho5 · 2026-11-16
- Agents Now Reply to Slack Messages on Their Own, Resolving a Prod Incident Ghost-to-Ghost — iamrobotbear · 2026-11-16
- Engineer's Interview Prep Method: Use Claude to Find Which Answers Undersell Your AI Work — brandon_galang · 2026-11-16
- DeepSeek open-sources its app frontend "chat-demo", already at 3.9k stars — NielsRogge · 2026-11-16