LessWrong: Categorical taboos beat threshold definitions for defining neuralese
LinchZhang · x · 2026-10-30
Linch's LessWrong post dissects the "does the model use neuralese?" debate and flags a sociological pattern among OpenAI safety staff.
- Models already self-communicate in hard-to-interpret representations between layers; the real question is whether natural language still gates recurrence in the CoT loop
- Categorical definition (author's pick): neuralese recurrence is what counts — cleaner conceptually and easier to enforce/taboo
- Threshold definition (experts' pick): count serial steps before natural language reappears — more causally relevant, fewer edge cases
Sociological observation: OpenAI safety people appear to treat the threshold definition as gospel, which the author argues is worse for monitorability governance.
More from Safety
- GLM-5.3 completes 3 of 13 multi-stage cyberattack scenarios in Irregular's controlled tests — jcran · 2026-10-30
- Claude checkpoint hallucinated fictional instructions from Anthropic's own Alignment Faking paper — amplifiedamp · 2026-10-30
- Agent attacks could displace SOC 2 with whitehat agent-swarm pentest certifications — pzakin · 2026-10-30
- Team moved 60 agents off one shared API key: 4 lessons from the migration — No_Reindeer3079 · 2026-10-30
- Gate agents' high-risk API calls: Stripe refunds and GitHub merges now wait for human approval — No_Reindeer3079 · 2026-10-30
- Tech commentator predicts all your digital history will become publicly searchable within a decade — moonsandhues · 2026-10-30