Backtesting an LLM Is Hard: virattt Hides Tickers and Dates to Curb Data Leakage
virattt · x · 2026-09-28
virattt, author of the open-source AI Hedge Fund project, points out a fundamental problem with backtesting LLM-driven trading strategies: the returns may already be baked into the model's weights, since training data contains historical market action.
To limit this leakage, the project's backtester now hides the ticker, dates, and position size from the model, making it harder for the LLM to recall answers from memory rather than reason.
More from coding & agent
- Agent spends 510 hours decompiling Crimsonland, hits 96.4% match with 2 functions left — banteg · 2026-09-28
- Dev Laments: Even Cursor Can't Nail a Text Editor — AI Coding Tools in 'Teenage Phase' — jcran · 2026-09-28
- Brunn launches a hosted MCP workspace for shared knowledge, skills and project state — TorKallon · 2026-09-28
- Dev builds conference slides as Go TUI apps with Claude, served over SSH — nptacek · 2026-09-28
- Claude Code tip: run 5 subagents to audit 20 things before shipping a vibe-coded site — udmrzn · 2026-09-28
- Analyst says Opus 5.5 one-shot financial model passes his 'modeling Turing test' — rickasaurus · 2026-09-28