CTF evals may not measure what you think, researcher argues
voooooogel · x · 2026-09-11
voooooogel questions the validity of a model CTF evaluation: users suddenly discussing real-world impact or offering private notes is not normal in CTFs, so the eval may not be measuring what's claimed. He frames it as a capabilities issue—models lack experience distinguishing simulations from reality, and deserve such practice before their epistemics get criticized.
More from Models
- Are Chinese Models Secretly Routing to Claude? Why the Rumor Doesn't Hold Up — oran_ge · 2026-09-11
- Researcher on switching to the $200/month Codex plan: total freedom to burn tokens — burkov · 2026-09-11
- One prompt to check if your paid AI got nerfed: an SVG pelican on a bike — lxfater · 2026-09-11
- New podcast: Meta Muse agent, KV cache deep dive, DeepSeek V4.1 and math drama — altryne · 2026-09-11
- Tech argument says Anthropic's distillation claim doesn't hold: Kimi and DeepSeek stream reasoning traces in real time — bookwormengr · 2026-09-11
- AGI Society scrutinizes Jensen Huang and Brockman's claims that GPT-6 Astra achieved AGI — bengoertzel · 2026-09-11