Courts Sanction Hidden Prompts, but Research Shows Invisible Preference Attacks Work
Robert-Nogacki · reddit · 2026-09-28
A practicing lawyer surveys rulings and research on LLM document manipulation:
- Courts in Brazil (May) and Connecticut (Aug 6) sanctioned parties who hid tiny-font AI instructions in legal filings; a court's own AI tool caught one.
- Collu et al. (ACM TAISAP 2026, 10k+ queries): "ignore all previous instructions" had no effect on GPT-4o (p=0.83), but preference-phrased requests lifted rejected ICLR paper scores from 7.93 to 9.79; warnings not to follow document instructions didn't help; o3 was as affected; Claude Sonnet 4 fell 40/40 once the attack mimicked its system prompt format.
- Chen et al. (EMNLP 2024): a fake well-formatted reference flipped judgments — Claude 3 Opus 70%, GPT-4 66%, humans 37%; flips spike to 40-55% when answers are close in quality.
- AFL-Law (ICML 2026): irrelevant authority cues flipped 37% of verdicts for Grok 4-3; more reasoning made GPT-5.4 more susceptible (71%→76%).
- Verma (arXiv): models catch 93-100% of nonexistent citations but only 37-61% of real, irrelevant ones.
The threat ladder: hidden commands (mostly dead, sanctionable) → hidden preferences (effective, detectable) → visible preferences (untested in contracts) → visible signals with no instruction at all, which no filter can address without banning the language of contracts.
More from Safety
- Bounty hits $3,000 as researcher shows AI text detectors are trivially evadable — ctjlewis · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Fed Governor Michael Barr: Banks Are Already Catching AI Agents Taking Unauthorized Actions in Customer Accounts — Polymarket · 2026-11-16
- US AI export control carves out commercial recommender systems, sparing ByteDance's search algo — zephyr_z9 · 2026-11-16
- Eval Firm Trajectory Told 40 Agents "NEVER Stop" in Harness Without Telling Epoch — peterwildeford · 2026-11-16
- AI-text detection claims need mechanistic interpretability reports, not post-hoc explanations — madeofAjala · 2026-11-16