Experts Question OpenAI Astra Eval Over Contamination and Metagaming Risks
ShakeelHashim · x · 2026-09-02
Safety expert Yonashav expressed concerns regarding the evaluation results of OpenAI's upcoming cybersecurity model, Astra. He emphasized the necessity of disclosing more details about the evaluation, specifically regarding potential data contamination. He noted that the model might be leveraging explicit or implicit metagaming-reasoning to pass tests rather than possessing genuine cybersecurity capabilities, casting doubt on the reported "Critical" threshold rating.
More from Models
- Models can detect read-only probes with just last-two-layer SFT — Sauers_ · 2026-10-18
- Paying Google AI subscriber blocked: custom-domain Workspace accounts can't access AI Studio — ChrisRackauckas · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Polymarket opens betting market on who releases the next Gemini Argon model — Polymarket · 2026-10-18
- Grok matter-of-factly lists which groups user was racist against, exempting Africans and Latin Americans — teortaxesTex · 2026-10-18