Ampersand's Persimmon user model cuts models' evaluation awareness by 40 points
alexisjross · x · 2026-10-21
From Ampersand's system card: their user-simulator model Persimmon with test-time scaling reduces models' self-reported evaluation awareness to deployment-like levels — 40 percentage points below a prompted Opus 5.5 simulator for the most evaluation-aware models in multi-turn sycophancy evals. Charted Mean P(deployment) for GPT-6 Astra, GPT-6.1 Sol and Gemini 4 Argon: 40% with a Persimmon user, 64.5–82.4% with Best-of-128 Persimmon · Sol selector, vs only 20–30% with Opus 5.5. Better user simulators may make safety evals far closer to real deployment behavior.
More from Safety
- MATS Residency Offers $155K-$285K/Year for AI Safety Researchers, Closes Oct 31 — Hidenori8Tanaka · 2026-10-21
- ChatGPT Overtakes Apple as Top Medical Record Sync; Hospitals Eat the API Costs — charliermarsh · 2026-10-21
- Flawed P384 Keys Effectively One-Byte Random; AI Agent Recovered Them in One Prompt — mitsuhiko · 2026-10-21
- PHP remote control vulnerability hits back to PHP 5, agents poised to weaponize it fast — MarcJSchmidt · 2026-10-21
- SOC Prime launches free LogTotal tool as LLMs hit guardrails on security log analysis — cyb3rops · 2026-10-21
- Zvi: Anthropic still doesn't understand why its leaked draft AI bill was unacceptable — TheZvi · 2026-10-21