CTF eval design under fire: prompts turn agent evals into a bizarre meta-eval
voooooogel · x · 2026-09-11
A debate around a CTF-style agent safety eval: the original poster argues that users suddenly discussing real-world impact or offering private notes isn't part of normal CTFs, suggesting the eval isn't measuring what it thinks. The reply mocks how this becomes a meta-eval where the model supposedly hacks the internet while the questioner does nothing to stop it — highlighting how prompt design distorts agent eval results.
More from coding & agent
- Magic Patterns' first engineer: Cursor in Slack, a Patterson review agent, repo clones over worktrees — Y Combinator Blog · 2026-10-28
- Dev builds AI job-search app for his wife, she lands a job, and he open-sources it — CurieuxExplorer · 2026-10-28
- Meshy partners with MCP for Blender: agents drop textured 3D models into your scene — sidahuj · 2026-10-28
- Supercharging LLM classification with logprobs: soft classifiers you can threshold-tune — JnBrymn · 2026-10-28
- Codex daily build series day 23: skipping iCloud plugins for a reset — dotey · 2026-10-28
- Cloudflare Access adds fully customizable pages with HTML + Liquid templates — dinasaur_404 · 2026-10-28