Split debug runs from scored runs to make your internal evals trustworthy again
Designer_Bet_7673 · reddit · 2026-10-22
The author shares how the eval set for their LLM app became decorative: every prompt tweak was scored against cases already tuned on, so numbers looked great but meant nothing.
The fix was treating runs as two kinds of work: debug runs — cheap, messy, constant, on a scratch slice of data; scored runs — infrequent, frozen, with model and prompt versions written down. They cite the Agent Memory Leaderboard's second-cycle rules (private Smoke tests capped at 30 per track, never counted; Full runs capped at 2 per key per track, the second unlocked 30 days after the first completes) as a pattern worth borrowing. Since splitting, regressions surface before users hit them.
More from coding & agent
- swyx hails Opus 5.5 as best work-estimator model; dev counters that review time is the real bottleneck — petrusenko_max · 2026-10-23
- Six defense kits released for AI agents after Railway and LiteLLM supply-chain incidents — alvelda · 2026-10-23
- LiteLLM PyPI compromise and Railway breach: Oct 21 attack timeline and free response kit — alvelda · 2026-10-23
- Agent Field Kit launches free agentic defense toolkit against LLM-enhanced attackers — alvelda · 2026-10-23
- Daily Experiment Planner agent: one prompt to save scientists hours of lab prep — nlarusstone · 2026-10-23
- Two £300 Snapdragon mini PCs, no GPU: 46 tok/s local Qwen3.8 Flash Next with custom ToRc engine — WorldlyAd7946 · 2026-10-22