Pass/fail scores no longer interpret evals: overly strict hidden tests create false failures
xhluca · x · 2026-09-12
xhluca argues that it's basically impossible to interpret evals from pass/fail scores alone these days: many benchmark failures come from overly strict hidden tests, where the model's answer is sometimes more sensible than the expected eval result. A replier adds that on non-binary problems, anything above 95% may now be an anti-signal.
More from Models
- Omni-InteractScope benchmarks proactive interaction in continuous omnimodal streams — Qwen · 2026-10-28
- Mistral releases Timestral time-series foundation models, tops GIFT-Eval — mistralai · 2026-10-28
- Gemini 4 Argon Hits 1.73% WER on OCR Benchmark, Surpassing Human Experts — gabriberton · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28
- Uncensored LFM3-1.7B GGUF weights released on Hugging Face — kalyan_kpl · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28