"I Thought You Were The Uncensored Place": Norms, Rules, and Moderation in AI-Generated Sexual Content Communities
Lucy Qin, Jaron Mink, Elissa M. Redmiles
cs.CR
2026-08-14
Interviews with 24 members and moderators of three 10,000+ member Discord communities for AI-generated sexual content find rules targeting shared outputs while the asset and knowledge infrastructure that enables abuse stays unregulated, as anti-censorship norms leave moderators justifying enforcement only via law and terms of service.
Open-weight image models (SDXL, Flux), model repositories like CivitAI, and jailbreaks let anyone generate nearly any form of sexual content. Mainstream AI platforms responded by banning all of it, which pushed creators toward self-organized Discord communities. Most of these communities publicly prohibit non-consensual imagery and content involving minors, but no empirical work had examined whether their governance actually holds. A Georgetown and Arizona State team interviewed 24 members and moderators between April and August 2025 across three large communities (A and B image-focused, 10k-20k and 50k+ members; C text-focused), all with stated rules against abusive content.
Qualitative interview study. The authors identified active Discord communities with explicit rules against abusive content, obtained moderator approval before recruiting, and interviewed 2 to 14 members per community at 40 USD each. Analysis used reflexive thematic analysis: independent coding, merged codebooks, triple-coded transcripts, then division of the remainder. The framing draws on group formation theory and Ostrom's institutional analysis, yielding four questions: how communities form, what norms members hold, what rules exist, and where governance fails.
The ethics apparatus was heavy: the team presumed some participants might be offenders, set up limited confidentiality per established guidelines for interviewing sexual-violence offenders, and prepared termination and reporting procedures. Three participants disclosed creating non-consensual imagery during interviews; one was subsequently found to have publicly posted content depicting an identifiable victim, which was reported and removed.
Three norms ran through every interview:
Defenses exist at both levels. Most participants held a personal commitment against underage content, fewer against non-consensual imagery; all three communities had public rules against sharing underage content, and that rule overrode the other norms. The gap: rules cover outputs, not infrastructure. Apart from text-focused Community C banning jailbreaks and prompts that could enable underage content, essentially no community governed the circulation of assets, datasets, or knowledge. Community C, unable to control how shared jailbreaks get used, ended up writing its own more restrictive ones to distribute.
The enforcement picture follows. Judging a generated character's age is eyeball work, so moderators err on the side of caution around school settings and uniforms; telling whether a real person in an image was face-swapped, original, or consenting is often impossible; at scale, moderators reported the server needs eyes every ten minutes. In a community whose identity is anti-censorship and non-judgment, the only legitimizing ground for removal is law and platform terms; Community A's moderator quoted members pushing back with "I thought you were the uncensored place" and settled on legality as the line. Content with clear criminal liability is a hard boundary, while non-consensual imagery lacks equivalent legal clarity in most jurisdictions, leaving moderators unable to articulate why a takedown is justified.
The paper also flags a structural risk: these communities are highly visible, searchable, and rarely deplatformed, while accumulating resource infrastructure (data, tutorials, custom tools) that closely mirrors abusive communities. Members with firm personal limits still absorb the framing of abuse as an inevitable byproduct of the technology.
Model-level filtering cannot reach locally deployed open weights, and platform crackdowns push users toward more extreme corners, so community governance is the remaining intervention surface. The findings serve three audiences. Detection: distinguishing lawful creation from abuse requires recognizing real people's likenesses in generated content, an open problem that nudity detection does not solve. Policy: current US law mostly penalizes outputs and distribution, while a recent UK amendment criminalizes commissioning requests; the paper supports expanding regulation toward the enabling ecosystem of assets, commissions, and tutorials. Community tooling: volunteer moderators lack corporate moderation resources and need sociotechnical support, not more deplatforming.
The authors disclaim generalizability: only communities with stated rules, whose moderators consented to research, were selected. Knowing offenders rarely volunteer, yet three still disclosed offending, suggesting the true abuse surface is wider than the sample shows. Qualitative work carries no effect sizes; the claim that unregulated assets enable abuse is a mechanism argument without quantification. Everything rests on self-report, and moderators' accounts of enforcement could not be verified.