Your LLM eval is probably lying: 4 failure modes and cheap fixes from an Istanbul talk
irombie · x · 2026-10-30
A thread from the author's AI Connect Istanbul talk lays out why LLM evals mislead and the cheap habits that catch each failure:
- Benchmarks are saturated and contaminated: nearly half of 60 studied benchmarks are saturated, and OpenAI stopped reporting SWE-bench Verified after frontier models reproduced the gold patches. Fix: build 20 examples from your own failures.
- LLM judges are cheap but biased: $3.60 vs $500 per 1,000 outputs vs human labels, with position, verbosity, and self-preference bias. Fix: validate against your own labels, run every pair in both orders, treat a judge swap as a new eval.
- Agents calling the right tools can still fail: correct answers can come from loops burning tokens. Fix: score the whole trajectory — tool correctness, task completion, step efficiency, plan adherence.
Related event: Your LLM Evaluation May Be Lying: High Variance and Contaminated Benchmarks(3 posts)→
More from coding & agent
- A 4B Model Verifies a Coding Agent 70x Its Size at Just 4% of Steps, 9-13% Extra Cost — petrusenko_max · 2026-10-30
- Static RAG eval sets go stale: treat your eval dataset as a pipeline, not a file — paulabartabajo_ · 2026-10-30
- What took a week now takes ~5 minutes with Sol or Opus coding agents — D3VAUX · 2026-10-30
- dhh: 'Review every line' is AI psychosis — the future is verifying outcomes, not code — CtrlAltDwayne · 2026-10-30
- Do you need an AI QA agent writing Playwright tests when AI already reviews PRs? — Infinite-Fly-2756 · 2026-10-30
- Octoverse: Claude Code tops fastest-growing repos; helium lands at #9 — uwukko · 2026-10-30