The eval gate worth copying: held-out validation beats clever prompts
krishnan · x · 2026-11-11
The author breaks down a rigorous evaluation design: a new solution only counts if it beats the incumbent on held-out validation data by a noise margin, and the final score comes from a hidden test split used exactly once. That gating mechanism—not the clever prompts—is the part worth copying.
What stays fixed matters just as much: the evaluated model's weights never change (some solutions train helper models, but no model rewrites its own parameters), and the evaluator, data splits, and budget accounting are all locked—the agent cannot touch the grading.
More from Research
- Infant brain 3D viewer goes official, adds MCP server for Claude/ChatGPT/LangGraph agents — Dr_Alex_Crimi · 2026-11-11
- Pretraining insider notes: data repetition and ~1000 TPP overtraining are standard practice — SonglinYang4 · 2026-11-11
- Sliding Window Attention Explained with Cartoon Huangs: Old Tokens Slide Out of View — cfregly · 2026-11-11
- Baseten details deep research agent training: breadth-rewarding RL plus SFT for report writing — baseten · 2026-11-11
- onóna: twin embeddings with tied unembeddings cut validation loss across scales — Cheng Luo · 2026-11-11
- Caveats: small models only, and 50k+ poison examples can work without clean data — OwainEvans_UK · 2026-11-11