Nested walk-forward vs the Optuna Sharpe: how HPO quietly overfits your quant backtest
PtrPomorski · x · 2026-09-27
A common quant ML trap: wire Optuna to a strategy Sharpe, get a glossy number, and believe you optimized the model — when you actually optimized the backtest itself.
The author built a deliberately boring experiment to measure the gap:
- Data: SPY daily, 2005-01-03 to 2026-09-18 (5,397 feature rows); target is next-day return regression, features strictly as-of close with no lookahead
- Model & signal: sklearn HistGradientBoostingRegressor; long when prediction > 0, else flat
- Features: 12 vanilla factors (multi-horizon returns, vol, momentum, z-scores, RSI) — the kind people still ship
- Costs: 5 bps one-way per position change (10 bps round-trip), charged inside every Sharpe that is optimized or reported, not sprinkled on afterwards
- Comparison: naive-path Optuna TPE (35 trials) vs nested walk-forward (25 trials per inner loop), benchmarked against SPY buy-and-hold over the same dates
Takeaway: the naive-path Sharpe routinely looks too good; only cost-aware nested out-of-sample results are a trustworthy headline number. The methodology — not the feature zoo — is the story.
More from Research
- Claude Code and Codex Beat Human Speedrun Records, But No Model Invented a New Optimizer — AI Engineer · 2026-09-27
- Anthropic launches a wet lab where AI agents and scientists run biology experiments together — marinkazitnik · 2026-09-27
- 421M fine-tuned model Gyra v0.2 catches destructive coding-agent commands, ships as Claude Code hook — CelebrationAble6568 · 2026-09-27
- Free Open-Source Site Lets You Deep-Read Einstein's Four 1905 Papers — doodlestein · 2026-09-27
- 119k papers later: top-venue acceptance exploded from 968 to 19,003 per year since 2006 — angelaqdai · 2026-09-27
- GEPA reflection beats RL: 3 examples outgain GRPO's 25k rollouts, ARC-AGI to 89.5% — AI Engineer · 2026-09-27