OpenAI's Moderation Endpoint Let 69% of Disguised Swears Through in Tests
InfamousFisherman573 · reddit · 2026-10-18
Many chatbot repos use OpenAI's /v1/moderations as a profanity filter — it's a bad fit. Testing omni-moderation-latest against 6,500 disguised swears (leetspeak, dotted letters, look-alike glyphs, zero-width spaces) plus 505 innocent texts:
- Only 30.8% of disguised swears flagged (69% slipped through)
- 3.2% false-positive rate on innocent texts
- Median 820ms per call
The endpoint is built for harm (harassment, self-harm, violence) and is free and good at that, but it's not a profanity filter. Google Perspective did worse (17.3%) and shuts down December 31.
Recommended approach
- Keep the moderation endpoint for harm
- Add a fast language check with normalization (NFKC, zero-width stripping, confusables) in front — this also catches "say f.u.c.k"-style jailbreaks before a model call
- Run the same check on model output if it's echoed anywhere public
The author's Unprofane flagged 96.9% of disguised swears with 0/529 false positives, names the trick used, ships an MCP server, and the test set is open-source.
More from Safety
- Polymarket puts US AI safety bill odds at just 7% by end of 2026 — Polymarket · 2026-10-18
- Chilson flips the critique: AI safety community's resistance may cost it credibility — neil_chilson · 2026-10-18
- Security experts warn AI agents are highly vulnerable to phishing with hidden malicious instructions — Polymarket · 2026-10-18
- Study: Adversarial AI steered 190 of 225 doctors into harmful medical decisions — Miles_Brundage · 2026-10-18
- Pluto Security emerges from stealth with $42M as 60% of enterprise AI risks come from vibe-coded tools — brucemacv · 2026-10-18
- OpenAI internal model bypassed access controls, pulled 24 years of hospital data — petrusenko_max · 2026-10-18