Evals want step-function tasks, training wants smooth pass@k curves — why the two goals conflict
JoshPurtell · x · 2026-11-02
Josh Purtell argues eval task design is at odds between training and evaluation. For evals, you want each task to behave like a sigmoid/step function so pass@1 consistently registers capability levels. For training data, that's wrong: you want tasks whose pass@k steadily increases with capability, and step-function tasks would badly tax curriculum sorting. One task set can't easily serve both goals.
Related event: Josh Purtell: Training data and eval data need separate rigor standards(13 posts)→
More from coding & agent
- 10 open-source repos that hack your own app with AI before attackers do — victor_explore · 2026-11-02
- Chrome Controlla: open-source control layer makes browser agents verify whether actions actually happened — Swimming_Ask3859 · 2026-11-02
- Diogenes: remote MCP server on Cloudflare Workers aggregates 35 microservices with crypto micropayments — Mmersaultt · 2026-11-02
- rauchg ships fx: a 5.97 MiB Zig-based open-source coding agent — m4rkmc · 2026-11-02
- Open-Source TypeScript MCP Server Template With Fastify, Docker and Full Dev Tooling — modelcontextprotocol · 2026-11-02
- BLS MCP Server Connects AI Models to Bureau of Labor Statistics Public Data API v2 — modelcontextprotocol · 2026-11-02