Distillation exploit reported in May went unpatched; OpenAI and Anthropic later disclosed campaigns
GarrisonLovely · x · 2026-10-28
Garrison Lovely lays out the distillation exploit timeline:
- US labs hide chain-of-thought scratchpads to deter distillation, but don't seem to treat it as a serious threat
- In May, security researcher Matthew Green found an exploit that could expose raw chain of thought; he reported it to OpenAI and Anthropic, which chose not to patch it
- In August, other researchers published a paper turning the idea into a working exploit; by September, both Anthropic and OpenAI disclosed coordinated distillation campaigns using it
- Anthropic's head of safeguards told a researcher: "Man we've seen so much traffic that uses attacks you described in that paper!"
Months elapsed between known vulnerability and large-scale exploitation.
More from Models
- Tweak one neuron and Qwen3.5-4B turns into a screaming oracle: PEFT activation steering demos — LysandreJik · 2026-10-28
- Surge AI at Kaggle Bench Summit: why frontier models still struggle with work-like benchmarks — echen · 2026-10-28
- DeepSeek rerun through Fireworks reproduces results; original dip was likely a fluke — teortaxesTex · 2026-10-28
- Subscription-subsidized cost chart shows DeepSeek V4.1 Flash hitting the frontier for free — teortaxesTex · 2026-10-28
- 20,869 agent traces show frontier models cut failed repair loops by 6.9 points — arena · 2026-10-28
- Nemotron-3.5-Transcribe lands native vLLM integration with FP8/NVFP4 and speculative decoding — aakashgupta · 2026-10-28