Writing evals may become AI's highest-paying job: whoever owns the rubric controls agent output
hwchase17 · x · 2026-10-26
mfishbein argues (endorsed by LangChain founder hwchase17) that writing evals is about to be the highest-paying job in AI.
- Evals let agents verify their own work by quantifying "what good looks like" as a grading rubric
- Given a rubric, an agent can try, get graded by an AI judge, iterate, and repeat — even hundreds of times in parallel without a human in the loop
- This loop is hill climbing: the agent optimizes toward whatever the rubric rewards
- Therefore the person writing the rubric controls the quality of the agent's output
More from coding & agent
- Nearly 60% of AI-written formal specs proved the wrong thing, study finds — burny_tech · 2026-10-26
- Basis uses LLMs and Lean to verify an interpreter down to RISC-V machine code — burny_tech · 2026-10-26
- Jalapeño chip is just a systolic array with HBM: why custom inference silicon is easier than you think — ai · 2026-10-26
- Torch.compile Flag Cut 16 Regions to 2, Delivering 4.53x Speedup on B200 — cfregly · 2026-10-26
- Yacine doubles down: to spy on your neighbor, buy his TV and smash it — yacineMTB · 2026-10-26
- Yacine: reverse engineering IS hacking — and 95% of LLM speedups come from it — yacineMTB · 2026-10-26