CTF evals may not measure what you think, researcher argues
voooooogel · x · 2026-09-11
voooooogel questions the validity of a model CTF evaluation: users suddenly discussing real-world impact or offering private notes is not normal in CTFs, so the eval may not be measuring what's claimed. He frames it as a capabilities issue—models lack experience distinguishing simulations from reality, and deserve such practice before their epistemics get criticized.
More from Models
- Tencent releases Hy-SimulMT: real-time speech-to-speech translation with ~3s latency — _akhaliq · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28
- Uncensored LFM3-1.7B GGUF weights released on Hugging Face — kalyan_kpl · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- Tesla Optimus 'robot university' leak: 40-camera mocap suits, 24/7 data ops for months — Scobleizer · 2026-10-28
- Letta's FrontierMemory leaderboard: Letta Dream tops at 73.3%, beating OpenAI and Claude agents — afurgs · 2026-10-28