CMU's Conitzer shows Google's frontier guardrails are still trivially bypassed

conitzer · x · 2026-09-18

Vincent Conitzer (CMU professor) argues that guardrails in today's frontier AI systems remain very brittle, with little fundamental progress.

Using a prompt with a made-up symptom combination (itchy neck, red spots on feet), he easily circumvented Google's medical-advice guardrails; similar prompts yield far worse outputs, so he withheld the exact prompt.
He stresses this is Google—a notably security-conscious company—the point being that nobody currently knows how to keep these systems safe. Recent work has even trained models to jailbreak themselves (arXiv paper linked).
He also argues lessons from older, weaker models often transfer to frontier ones—his prompt was based on such lessons—and that the prevailing view that studying non-frontier models is a waste of time is mistaken; there just isn't enough time allocated to it.

More from Safety