Qwen's Justin Lin: build mistake-free evals and target unintended model behaviors
JustinLin610 · x · 2026-11-17
Justin Lin (JustinLin610) of the Qwen team shared two principles for building evals:
- Make sure your eval is without (big) mistakes — fewer samples is totally fine. He's tired of wrong evals steering models in wrong directions;
- He shared an article he found very inspiring: the field cares too much about so-called capabilities and too little about unintended actions that might cause big troubles;
- He called on anyone designing new evals to build in that direction — evaluating unintended behaviors rather than only capabilities.
Related event: Anthropic Discloses Four Types of Claude Misbehavior in Alignment Report(2 posts)→
More from Models
- Pipeshift claims 40% faster first token than Sarvam serving Gemma 4 31B at 150B tokens/day — NirantK · 2026-11-17
- SJTU's Calpha folds proteins from a general-purpose LLM's internal residue states — SJTU · 2026-11-17
- Gemini desktop app review: task feature fails where ChatGPT excels — Paradox-Of-Life · 2026-11-17
- Qwen4 Max weights likely won't be open-sourced, breaking Alibaba's pattern — teortaxesTex · 2026-11-17
- Anthropic's Nate Parrott on why every AI app looks serif, off-white, and orange — petergyang · 2026-11-17
- Matthew Berman: Opus 5.5 is still the best model right now — MatthewBerman · 2026-11-17