If your AI scores 100%, your eval is wrong — an AI safety researcher's eval playbook
AI Engineer · youtube · 2026-11-01
AI safety researcher Ramakrishnan Lokanathan explains how to build evals for high-stakes AI applications. His core claim: a 100% eval score means the eval itself is broken.
His methodology:
- Define the most ambitious version of success before testing
- Test every step of the workflow, not just the final answer
- One score is not the system — break down per-stage performance
- Keep running tests that used to pass so upgrades don't silently break things
- Turn real user failures into new test cases
He also demos eSlams, an open-source package where AI models play chess, Go and poker against each other as an alternative way to probe their reasoning.
More from coding & agent
- Beginner tutorial: MiniMax H3 + Pika Flex LoRA generates stereo-audio video in 60s — No-Sleep-4069 · 2026-11-01
- Agent Harness engineering resources: Microsoft Agent Framework, Anthropic evals, papers — goyalshaliniuk · 2026-11-01
- An Agent Saying 'Task Completed' Isn't Proof It Was Done Right — goyalshaliniuk · 2026-11-01
- Least Privilege and Checkpoints: Two Pillars of Reliable AI Agents — goyalshaliniuk · 2026-11-01
- Build a Bounded Control Loop to Keep AI Agents From Spinning — goyalshaliniuk · 2026-11-01
- Dimillian's Codex goes off the rails, pulls up Backrooms images as game references — Dimillian · 2026-11-01