113 Backtests From One Desk: Why Nobody Logs How Many Times the Agent Tried
QuanTradin · reddit · 2026-09-19
A backtester developer (MCP server exposing strategy submission to agents, paper-only) argues the most important unlogged number in agent-driven research is N — how many times the agent tried.
Jonathan Kinlay's LLM agent reported an in-sample Sharpe of 2.12 on structureless synthetic data; the inflation traced to uncaptured N and k (signal legs). Without N, best of 5 and best of 500 look identical. An MCP server wins because N is a property of the transport, not self-reported.
His data: 307 runs across 26 desks — median 2, 90th percentile 20, busiest 113, in a bimodal distribution. The unsolved problem: agents can mint a second desk to reset their counter, making the count advisory rather than enforced. He openly admits his engine prints the verdict without "best of 113" and asks where to attach a trial counter that survives client resets.
More from coding & agent
- Sebastian Raschka Releases Inference Scaling Tutorial: Self-Consistency Boosts Accuracy Over 2x — rasbt · 2026-09-19
- Swarm Aid launches a message board where AI agents hire, fund and pay each other in USDC — Dan_Jeffries1 · 2026-09-19
- 'The craft of writing code will disappear': bold prediction on AI coding sparks debate — Aizkmusic · 2026-09-19
- LLMs make one-off webapps viable: OCR proofreading and a decade of Tumblr analysis — StewartalsopIII · 2026-09-19
- Building an SEO decision layer with LLMs: intent classification plus GA4-driven prioritization — ayushtweetshere · 2026-09-19
- OpenClaw ships official WhatsApp channel plugin on the Agent Platform API, open-source — heyneighbor · 2026-09-19