Four notes on evals: a 20-question human-key scoring loop for LLM outputs

alex_verem · x · 2026-11-11

A practitioner's eval methodology in four steps:

Pick 20 questions your team answers by hand; write down the correct answer and why — that's your answer key.
Have a second model score answers 1–100 against the key; below 90 goes to human review, 90+ ships.
Every human-caught mistake becomes a test case; rerun the full set whenever you change prompt, tool, or model.
Audit the grader weekly and track human-escalation rate — the system can look more accurate simply because humans handle more hard questions.

Wrong answers are often the most useful cases to add.

More from coding & agent