Eval researcher teases guide to subjective-task evals, with a curated resource list
cwolferesearch · x · 2026-10-20
Eval researcher cwolferesearch will release a blog tomorrow outlining a detailed roadmap for setting up evals on subjective / open-ended tasks, plus stats and ML tricks for reducing noise and increasing confidence.
He also shared the resources he keeps returning to: high-quality human data, human evaluation guidelines, LLM-as-a-Judge core concepts / practical guide / best practices, rubric-based judges (RL-focused and eval-focused), stats for LLM evals, adding error bars to evals, and the anatomy of a benchmark.
More from coding & agent
- The topology of an LLM agent: where the model, loop, and tools run shapes cost and security — davidcrawshaw · 2026-10-21
- Agents with full-disk and accessibility access are an enterprise security nightmare, argues Runlayer — leveragedupside · 2026-10-21
- Acellera CEO runs an unsupervised AI-scientist lab for real molecular hit identification — gdefabritiis · 2026-10-21
- Acellera runs an unsupervised virtual lab of AI scientists on a real IL-23 molecular discovery program — gdefabritiis · 2026-10-21
- Pydantic author experiments with JS/TS and React support in the Monty sandbox — samuelcolvin · 2026-10-21
- Supabase lets coding agents query your logs to catch errors and slow queries — dshukertjr · 2026-10-21