Eval task design is a bias vs variance tradeoff, argues eval engineer
JoshPurtell · x · 2026-11-02
Continuing his thread on eval task design, Josh Purtell frames the training-vs-eval conflict as a bias-variance problem: variance is good and expected in training, but in evaluation it complicates analysis unless you have huge samples, which bring issues of their own. For evals, you'd generally prefer biased, low-variance results.
Related event: Josh Purtell: Training data and eval data need separate rigor standards(13 posts)→
More from coding & agent
- 10 open-source repos that hack your own app with AI before attackers do — victor_explore · 2026-11-02
- Chrome Controlla: open-source control layer makes browser agents verify whether actions actually happened — Swimming_Ask3859 · 2026-11-02
- Diogenes: remote MCP server on Cloudflare Workers aggregates 35 microservices with crypto micropayments — Mmersaultt · 2026-11-02
- rauchg ships fx: a 5.97 MiB Zig-based open-source coding agent — m4rkmc · 2026-11-02
- Open-Source TypeScript MCP Server Template With Fastify, Docker and Full Dev Tooling — modelcontextprotocol · 2026-11-02
- BLS MCP Server Connects AI Models to Bureau of Labor Statistics Public Data API v2 — modelcontextprotocol · 2026-11-02