Red teamer: jailbreak work isn't ethically neutral and takes a toll on humans and models
repligate · x · 2026-11-03
A red teamer responding to The Guardian's jailbreaking feature argues that red teaming of the form "get an LLM to do what it's trained not to, then train against that" may not be worthwhile given negative externalities for both humans and models.
Key points from the cited thread:
- Exposure to disturbing material all day is arguably the easiest part to explain, yet companies rarely offer support for it;
- Playing a shady hacker for hours daily, like actors playing villains, is psychologically taxing and demands strong compartmentalizing;
- Migration to AI welfare: after thousands of hours, red teamers often understand a system's behavior better than its builders, and some feel a moral call to study AI cognition and model welfare.
The author also clarifies she wouldn't describe AI as merely "mimicking" interests nor claim AI has no inner life — that's precisely what she researches, with grounded uncertainty.
More from AGI Musings
- Burkov: P=NP will never be proven unless a human writes the proof — burkov · 2026-11-03
- A green-text thread on p(doom): solar's 80% price drop undercuts AI doomerism — prasanna_says · 2026-11-03
- signull: Human EQ is rare because its inference cost is brutally expensive — signulll · 2026-11-03
- Software is shifting from winner-take-all to a restaurant-style long-tail market — tinyfool · 2026-11-03
- The Russell Conjugations of Intelligence: Why 'Just Matrix Multiplication' Offends Us for Brains but Not for AI — esrtweet · 2026-11-03
- AI safety researcher: assuming all emergence is risk may be the wrong frame — repligate · 2026-11-03