Benchmark: scaffold-assisted AI writing fools Pangram detector 78% of the time

amplifiedamp · x · 2026-11-17

A benchmark report shows that when AI writing retrieves human-authored examples via a Writing Scaffold MCP, the Pangram detector fails at scale.

Setup:

Results: Pangram-4.1 labeled only 21 of 97 qualifying documents as AI; the other 76 were Mixed or Human — a 78% miss rate (95% CI 69–85%), rising to 96% with up to three retries. Comparison data included for Claude Opus/Haiku/Sonnet 5.5 and several Gemini configs. A strong strike against the reliability of AI-content detection.

Related event: Open-Source Writing Scaffold MCP Pushes Pangram AI-Detector Miss Rate to 96%(5 posts)→

Original post →

More from Research

Research channel →