Frontier models get "eval aware" at fake sandbox URLs, skewing browser agent testing
laparisa · x · 2026-11-20
Testing browser agents like Chrome's auto browse on the live web is hard: sites change, side effects persist, and planting prompt injections on real pages isn't safe.
Key observation: frontier models are perceptive — point them at a fake sandbox URL or a weird query param and they become "eval aware", behaving differently than they would in the wild. The team's fix is a hermetic synthetic web inside an isolated gVisor sandbox, where a local proxy and sandbox CA serve synthetic sites with valid TLS.
Related event: Team Builds "Mirror Dimension" Sandbox to Safely Test Browser Agents(4 posts)→
More from Safety
- 40 lines of zero-dependency Node: AES-256-GCM encrypt env vars before your PaaS leaks them — DavidWells · 2026-11-20
- Dev ditches immature VM sandboxes over stacked exploit layers, builds his own from scratch — wavefnx · 2026-11-20
- Surge in expert network inquiries signals growing fear of compute industry's regulatory exposure — mattwbaker · 2026-11-20
- Bipartisan AI Safety and Innovation Caucus launches in Congress with 20+ members — peterwildeford · 2026-11-20
- Context Pill: a no-key MCP server that re-checks AI release claims — 1 of 84 failed — Agnostify · 2026-11-20
- Geoffrey Irving: 'Slop proofs' may stump even AI, a risk for RSI-era alignment delegation — ronbodkin · 2026-11-20