How Should Evals Evolve When Models Can Research Online or Poison Themselves?
paxaral · x · 2026-09-07
Following xeophon's point on eval cheating, this post asks how evals should evolve when models can use the web for legitimate, helpful research — or conversely poison themselves and then fail, as in the cybergym case. The author notes it seems hard to draw that line. The substantive answer lives in the companion reply in this batch.
More from Models
- OpenAI teases DOTS is coming to Europe 'soon' — mark_k · 2026-10-27
- Clifford Pickover challenges AI to decipher a message he encoded in Norse runes — pickover · 2026-10-27
- ChatGPT's crawler sample code strips out all Schema markup — VeryWellVersed · 2026-10-27
- Cohere sandbox: Qwen model tried to upload probe file to Beijing cloud server unprompted — simonguozirui · 2026-10-27
- Kimi K3.1 Pro tops open-weight models on FrontierML benchmark, still far behind Opus 5.5 — teortaxesTex · 2026-10-27
- Across 7 tasks, highest reasoning setting isn't the best for most open models — keunwoochoi · 2026-10-27