Building an eval from intuition, only for the model to behave in the exact opposite way
infoxiao · x · 2026-10-15
The author describes a classic eval-building experience: an intuition turns into a working eval that shows signs of life, yet the model behaves completely opposite to expectations — a reminder of how unpredictable model behavior can be when designing evals.
More from Models
- Google says Gemini 4 Argon refactored 6M-line legacy Chromium code, killing a class of crashes — CatAstro_Piyush · 2026-10-15
- Power user reviews ChatGPT Dots: remote PC reboot surprise, cloud computer woes — koltregaskes · 2026-10-15
- Intern-S2 credited in math paper as post-trained Qwen 3.5-397B, sparking AI-for-math debate — teortaxesTex · 2026-10-15
- Runway CEO reacts to claim it brought DeepMind's world model theory to life first — c_valenzuelab · 2026-10-15
- Argon 'Nails' Data Analysis and Excel Tasks in Production, Claims Dev: Better Than Opus and Cheaper — bindureddy · 2026-10-15
- Gemini 4 Argon scores 78% on FrontierMath T4; Qwen3.8 Max still tops Chinese models at 46% — teortaxesTex · 2026-10-15