IBM Engineer on Trustworthy AI Evals: Calibrate LLM Judges, Check Agent Trajectories
TejasKumar_ · x · 2026-11-15
IBM AI engineer and Fluent React author Tejas Kumar joined DataTalks.Club to discuss building AI evaluations you can actually trust.
Key points:
- Evals can show a green checkmark while your agent is confidently wrong; evaluation must go beyond manual spot checks.
- Topics covered: sourcing evaluation cases, calibrating an LLM judge to agree with human judgment, refreshing datasets, and pinning severe failure cases.
- Judge the agent's trajectory, not just its final answer, to catch issues in the process.
- Also covered: retraining judges, human preference sample sizes, and faster judging techniques.
The episode ships with code, demos and slides — useful for developers working on agent eval engineering.
More from coding & agent
- 18 agents coding overnight on one Linux box: 'I'm the bottleneck now' — AIOSAI · 2026-11-15
- Antigravity Browser Agent Swarm Turns a Portrait Into Google Sheets Pixel Art — Saboo_Shubham_ · 2026-11-15
- CrommaCode launches: a gamified desktop terminal for watching Claude Code and Codex work — huangyun_122 · 2026-11-15
- MCP server + Ollama drives Final Cut Pro: 30-min video rough cut in under 5 minutes — Technovangelist · 2026-11-15
- lattedit launches free local-first browser LaTeX editor with a JSON compile API for agents — cneuralnetwork · 2026-11-15
- Community-built wrath-mcp harness makes weaker models fight better at lower cost — max_paperclips · 2026-11-15