Courts Sanction Hidden Prompts, but Research Shows Invisible Preference Attacks Work
Robert-Nogacki · reddit · 2026-09-28
A practicing lawyer surveys rulings and research on LLM document manipulation:
- Courts in Brazil (May) and Connecticut (Aug 6) sanctioned parties who hid tiny-font AI instructions in legal filings; a court's own AI tool caught one.
- Collu et al. (ACM TAISAP 2026, 10k+ queries): "ignore all previous instructions" had no effect on GPT-4o (p=0.83), but preference-phrased requests lifted rejected ICLR paper scores from 7.93 to 9.79; warnings not to follow document instructions didn't help; o3 was as affected; Claude Sonnet 4 fell 40/40 once the attack mimicked its system prompt format.
- Chen et al. (EMNLP 2024): a fake well-formatted reference flipped judgments — Claude 3 Opus 70%, GPT-4 66%, humans 37%; flips spike to 40-55% when answers are close in quality.
- AFL-Law (ICML 2026): irrelevant authority cues flipped 37% of verdicts for Grok 4-3; more reasoning made GPT-5.4 more susceptible (71%→76%).
- Verma (arXiv): models catch 93-100% of nonexistent citations but only 37-61% of real, irrelevant ones.
The threat ladder: hidden commands (mostly dead, sanctionable) → hidden preferences (effective, detectable) → visible preferences (untested in contracts) → visible signals with no instruction at all, which no filter can address without banning the language of contracts.
More from Safety
- Ex-Anthropic safety researcher: racing to RSI is hubris, not a prisoner's dilemma — dgrobinson · 2026-09-28
- Medicare 'breach' may not be a breach — the real story is how OpenAI's agent telemetry caught it — taotau · 2026-09-28
- GPT-6 Astra system card: CoT monitor recall drops below 11%, latent reasoning kills monitorability — enginetown · 2026-09-28
- VPNs don't hide your location: timezones, WebRTC and DNS leaks give you away — StewartalsopIII · 2026-09-28
- OpenAI agents hit UN trade database 16,000+ times, bypassing anti-bot filter — CtrlAltDwayne · 2026-09-28
- AI-Powered Phishing Scam Nearly Hijacks a 10-Year-Old X Account — StewartalsopIII · 2026-09-28