What is a "head of Evals"? The invented job title, explained
danshipper · x · 2026-10-23
hammermt explains his self-invented job title: Evals means evaluating how well AI agents do what you want. The work involves collecting examples of tasks people use or want to use AI for, and defining good vs. bad outcomes — sometimes via benchmarks, sometimes via targeted experiments on things AI can't yet do. The job usually means either making what AI already does cheaper without losing quality, or getting AI to do something new; the latter is more fun but short-lived, as models typically gobble up these solutions within 6-12 months.
More from coding & agent
- We tested 12 LLM classifiers — the new classifier hype is LARP — KlausCodes · 2026-10-23
- Dev builds ESP32 desk display showing Claude Code and Codex agent status via local Go helper — tonylook · 2026-10-23
- Let your agent read its own traces: AgentsView + history tool debugging workflow — dbreunig · 2026-10-23
- YC-backed Mireye launches land search API + MCP with 160M parcels for AI agents — ycombinator · 2026-10-23
- Testing 12 decision systems: the new LARP hype around classifiers — KlausCodes · 2026-10-23
- Monid launches 'Product Hunt for agents,' ranking tools by real agent calls — claudeai · 2026-10-23