Frontier models get "eval aware" at fake sandbox URLs, skewing browser agent testing

laparisa · x · 2026-11-20

Testing browser agents like Chrome's auto browse on the live web is hard: sites change, side effects persist, and planting prompt injections on real pages isn't safe.

Key observation: frontier models are perceptive — point them at a fake sandbox URL or a weird query param and they become "eval aware", behaving differently than they would in the wild. The team's fix is a hermetic synthetic web inside an isolated gVisor sandbox, where a local proxy and sandbox CA serve synthetic sites with valid TLS.

Related event: Team Builds "Mirror Dimension" Sandbox to Safely Test Browser Agents(4 posts)→

Original post →

More from Safety

Safety channel →