11 LLM eval methods AI engineers must know, organized by what they measure
blaizedsouza · x · 2026-09-12
akshaypachaar shared a long thread covering 11 LLM evaluation methods, arguing there is no single metric — the right eval depends on what you're measuring.
Highlights:
- Reference-based evals when ground truth exists: BLEU (n-gram precision overlap), ROUGE (recall of reference content), BERTScore (contextual embeddings, so semantically similar answers score well)
- Other categories cover semantic correctness judging, multi-step agent behavior inspection, and blocking unsafe outputs before they reach users
- Paired with his open-source eval tool Opik (comet-ml, 22k stars) and an article titled 'Your Agent Harness Should Repair Itself'
More from coding & agent
- Frontier model for planning, local Qwen for coding: a hybrid dev workflow experiment — kirisoraa · 2026-09-12
- Codex + Astra xhigh tackle MazeBench: static BFS solver nets 44 gems in ~2 hours — xeophon · 2026-09-12
- Real2sim first, then real: user reproduces Astra controlling a robot arm to paint — ZeYanjie · 2026-09-12
- Eight years of logging electronic parts in .txt files now feeds AI project context — debreuil · 2026-09-12
- Worktrunk, a Rust CLI for Git Worktrees Built for Parallel AI Agents, Hits 7k Stars — max-sixty · 2026-09-12
- Claude-Red: Open-Source Red-Team Skill Library for Claude Hits 3.3k Stars — SnailSploit · 2026-09-12