Don't Let the Agent Grade Its Own Homework: What a Backtest Tool Should Return
QuanTradin · reddit · 2026-10-16
The indie developer of Quantradin raises a core agent-engineering question: the model drafts trading rules, but grading must happen in an engine it can't touch (no look-ahead, options filled at tradable bid/ask, not midpoint), returning a LIVE/SHADOW/SHELF verdict. The hard part is what the tool returns:
- Full trade list: burns context, and the model cherry-picks good months to argue with the verdict
- Verdict only: the model can't tell why a rule failed, so it nudges parameters until something passes — overfitting with extra steps
- Current lean: verdict + one or two driving numbers + paging through trades; but paging may bring cherry-picking right back
The post ends by asking the community: do you summarize on the server, paginate, or let the model request slices — and did the model get worse once it could see raw rows?
More from coding & agent
- Gemini 4 Argon goes live in Antigravity with its own weekly quota for Pro subscribers — rseroter · 2026-10-16
- LangChain revamps human-in-the-loop middleware with typed schemas for tool interrupts — Hacubu · 2026-10-16
- Robot fully controlled by frontier LLMs assembles gadgets autonomously for 1.5 hours error-free — ihorbeaver · 2026-10-16
- Why the future of agents is basically Node and JavaScript, in one argument — verrsane · 2026-10-16
- Vibe Coding a Data Pipeline Is Easy; Running It at Billions of Rows Is Not — Zachly · 2026-10-16
- How Claude Code Review works at GA: admin setup, auto reviews on every new PR — ClaudeDevs · 2026-10-16