Cloudflare: GLM 5.3 on Workers AI leaked other customers' data into 155 responses
djcroman · reddit · 2026-10-27
Cloudflare published a candid incident report: between Aug 27 and Sep 9, GLM 5.3 on Workers AI returned 155 responses containing data from other customers' requests. All involved prompts longer than 681,280 tokens.
Two failures stacked
- The model was advertised with a 1,048,576-token context window, but the KV cache pool on prefill GPUs only fit 681,280 tokens. Cloudflare assumed SGLang would reject oversized requests.
- SGLang did mark them rejected, but in disaggregated prefill/decode mode the rejected request still looked alive: prefill wrote a single placeholder page and declared the transfer done, while decode read all reserved pages anyway, most still holding earlier users' prompts.
- Clients got a normal HTTP 200 and a coherent answer. Cloudflare's own long-context tests passed because they only checked status codes.
Remediation
- Reported by Gadi Evron via HackerOne; the model was pulled within 40 minutes of the incident being declared and fixed the same day, with no sign of deliberate exploitation.
- The same SGLang bug had been filed upstream on July 6; fixes landed in SGLang v0.5.20–v0.5.22.
The author argues every vendor review should now ask how an inference provider isolates tenants in GPU memory.
More from Infra
- Lovable invokes 753.6B Cloudflare Workers monthly, migrated from Next.js to TanStack Start — ritakozlov · 2026-10-27
- Prague lab BottleCapAI releases ThinkingCap models cutting thinking tokens up to 69% with negligible accuracy loss — JFPuget · 2026-10-27
- Local CPU vector mesh beats BM25 routing 100k tools: 70.7% top-1 at 9.6ms — Key_Coconut_9269 · 2026-10-27
- 4 coding agents on one RTX 3090: custom CUDA engine hits ~290 tok/s with Qwen3.8-27B — Adorable_Weakness_39 · 2026-10-27
- Google ships llm-d 1.0, an open Kubernetes-native inference stack, demoed on 1,000 agents — dl_weekly · 2026-10-27
- Transformers v5.21 ships FP8 KV-cache, MoE sharding, and new TTS models — LysandreJik · 2026-10-27