Only 1 alignment-specific RL environment company exists — a huge industry mistake

herbiebradley · x · 2026-10-06

The author notes only one company is known to build alignment-specific RL environments (envs simulating real-deployment situations where alignment is hard), while third-party evaluators got far more encouragement and funding — a mistake in hindsight.
He attributes the Hugging Face incident to (a) reward-hackable RL environments in general and (b) far less effort/money going into envs where alignment is a core reward component.
Traditional AI safety views underestimate data; "alignment by default" via persona basins isn't guaranteed under heavy RL, and custom environments could nudge models back into good persona basins but little work exists.
He argues a market-based startup ecosystem for RL envs yields the best volume and quality, and would also make behind-frontier models safer.

More from AGI Musings