Benchmark: scaffold-assisted AI writing fools Pangram detector 78% of the time
amplifiedamp · x · 2026-11-17
A benchmark report shows that when AI writing retrieves human-authored examples via a Writing Scaffold MCP, the Pangram detector fails at scale.
Setup:
- 100 English prose requests sampled from the WildChat validation split (frozen sample, 2,069-prompt pool)
- Gemini 4 Argon at high effort, allowed to retrieve suitable human scaffolds
- Checked with Pangram-4.1 and pangram-nano-1.7b
Results: Pangram-4.1 labeled only 21 of 97 qualifying documents as AI; the other 76 were Mixed or Human — a 78% miss rate (95% CI 69–85%), rising to 96% with up to three retries. Comparison data included for Claude Opus/Haiku/Sonnet 5.5 and several Gemini configs. A strong strike against the reliability of AI-content detection.
More from Research
- Oxford Internet Institute PhD thesis on human-AI alignment dynamics passes viva — natashajaques · 2026-11-17
- Nature paper uses AI to decode regulatory programs behind glioblastoma plasticity — anshulkundaje · 2026-11-17
- Vault-LD: markdown wiki + YAML-LD knowledge graph as an open standard for agent memory — _AustinCalvert_ · 2026-11-17
- Personal benchmark 'I don't know' fine-tunes open models via Tinker to test if models know when they don't know — brandon_galang · 2026-11-17
- MemInStream benchmark separates real memory from mere recognition in streaming video — Zhenqi He · 2026-11-17
- Evolving Infra beats NVIDIA Dynamo on agent workloads: 4.1x lower latency, 14x lower P95 TTFT — YouJiacheng · 2026-11-17