Magic rebuilds knowledge evals per model generation to avoid eval overfitting
seanmcdonaldxyz · x · 2026-09-09
Magic, which builds models for SWE and autonomous AI R&D, explains its eval methodology: alongside its standard out-of-sample suite, it tracks "knowledge evals" and creates new ones for each model generation to avoid overfitting benchmarks over time. The approach sheds light on how frontier coding-model labs balance data trade-offs.
Related event: Magic Claims to Match DeepSeek V4 Pro Pretraining with 1/50 the Compute(11 posts)→
More from coding & agent
- AIE Code Summit 2026 opens applications: 800 seats, keynotes streamed free — trq212 · 2026-10-27
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Precompute golden answers and match at runtime for exact wording control — colinmcnamara · 2026-10-27
- Dev argues there's still alpha in working with your coding agent interactively — alex_frantic · 2026-10-27
- Researchers launch AMA on patterns for safer AI agents used by citizen developers — Loose_Rip359 · 2026-10-27
- Gradio Adds In-Chat Human-in-the-Loop So ML Intern Pauses Before Training — Gradio · 2026-10-27