Evals should be neutral, negative, or positive, argues an AI researcher
secemp9 · x · 2026-07-22
A simpler eval rule: neutral, negative, and positive evals
The author argues that eval design should be thought of in three buckets:
- Neutral evals: tasks that can be done naturally without the model being aware it is in an evaluation.
- Negative evals: setups where the model becomes eval-aware.
- Positive evals: eval-obliviousness, where the harness avoids giving the model hints that it is being tested.
The main point is that good evals should reproduce the task as it would happen in the real world, while avoiding prompts or harness design that telegraph the evaluation setting to the model. The thread uses Anthropic-related examples to argue that this issue is visible directly in the model's thinking traces.
Related event: Researchers Propose New Framework for AI Evaluation Design(3 posts)→
More from Research
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26
- Ortet evaluates 34 open-weight models on 50 health tasks, finds general leaderboards insufficient — elmanmansimov · 2026-10-26