Blog: OLMo 3's verbalized eval awareness is mostly copied from SFT training data
BlancheMinerva · x · 2026-11-11
A first blog post by yooniel examines verbalized eval awareness (VEA) in OLMo 3 and traces it to training data. Key finding: VEA — a model verbally signaling awareness of being evaluated — may not reliably measure actual eval awareness, since OLMo 3's VEA behavior is largely copied from SFT data.
Quoting author BlancheMinerva, who worked on T0, adds a related lesson: generalization after finetuning requires significant surface form variation. When training examples share similar presentation formats, models treat the format itself as causal to the output rather than learning true behavioral generalization — with direct implications for eval design and training data curation.
More from Models
- New model Carbon looks strong but runs 3x slower than Opus 5.5, says Bindu Reddy — bindureddy · 2026-11-11
- Nat Lambert: Jensen Huang leaving NVIDIA CEO role would badly hurt US open models — natolambert · 2026-11-11
- Gemini quietly renames models and appears to debut unannounced 4 Carbon and 4 Argon — Star-Light-9698 · 2026-11-11
- GPT Image 2.6 Extra High called unusable by Pro user amid influencer praise — StudyDemon · 2026-11-11
- Qwen3.8-27B uncensored finetune demoed doing real coding tasks on Ryzen AI Halo — LHAD16 · 2026-11-11
- 'Smell of burning copper': the LLM writing quirk that may trace back to DeepSeek R1 — aiamblichus · 2026-11-11