Benchmark: Pangram misses 78% of MCP-augmented AI writing, 96% with retries
amplifiedamp · x · 2026-11-16
amplifiedamp published a full false-negative benchmark of Pangram's AI-text detectors against writing augmented by the Writing Scaffold MCP server (report and GitHub repo public).
- 100 prompts sampled (seed 20261031) from 2,069 eligible WildChat English prose-writing prompts; Gemini 4 Argon high with the MCP server wrote every document
- Counting Mixed and Human as misses, pangram-4.1 missed 76 of 97 qualifying documents: 78% (95% CI 69–85%); only 22% were labeled AI
- An online variant regenerating on each AI verdict (up to 3 retries) pushes the miss rate from 78% to 96%
- Across 7 generator configurations, pangram-4.1 misses range from 32% (Claude Haiku 5.5 xhigh) to 79% (Gemini 4 Argon high)
The author fully released the method after encouragement from Pangram's Max Spero, judging the public benefit outweighs the harms.
More from coding & agent
- He skipped the Switch: Claude + Blender MCP turns a phone into a motion-controlled game console — stablequan · 2026-11-16
- Writing-scaffold MCP bypasses Pangram AI detector with 78–96% miss rates — amplifiedamp · 2026-11-16
- After Teaching 500-1000 People the Terminal, He Says LLM Skeptics Who Don't Build Should Be Doubted — StewartalsopIII · 2026-11-16
- mcp-openrussian: local MCP server grounds LLMs in a real Russian dictionary — No_Television_5967 · 2026-11-16
- Open-source Writing Scaffold MCP library helps AI writing bypass Pangram detector, up to 96% effective — amplifiedamp · 2026-11-16
- Datadog's new gateway returns 402 when a model's budget runs out—so agents just retry — YvesMulkers · 2026-11-16