Daniel Kokotajlo debates a Chinese AI safety startup on whether CoT monitorability evals prove enough
S_OhEigeartaigh · x · 2026-10-27
A notable cross-border exchange in AI safety: an emerging Chinese safety-eval startup discussed its CoT monitorability paper with former OpenAI researcher Daniel Kokotajlo (author of AI 2027).
- The paper tested CoT monitorability by (a) prompting models to manipulate their own chain of thought (they typically refuse) and (b) disabling reasoning to show models are far less capable without it
- Kokotajlo's critique: this isn't sufficient — what if the model didn't refuse and tried its hardest to manipulate the CoT? He'd like evidence of the form "even attempting its best manipulation, it still can't do it"
- Kristy Loke, who amplified the exchange, argues AI safety should be treated as an open science endeavor, not an agenda manipulated for self-gain
Related event: Chinese Eval Startup Debates CoT Monitorability with Kokotajlo(2 posts)→
More from AGI Musings
- Dario Amodei: he left OpenAI after losing trust in how safety disputes were handled — kimmonismus · 2026-10-28
- AI safety researchers: a few lab staff full-time could make US-China AI deal verification feasible — ohlennart · 2026-10-28
- ANT and OpenAI agents disprove ETH same day — are simultaneous AI discoveries inevitable? — yian_yin · 2026-10-28
- Palantir CEO Alex Karp: AI race is US vs China, with Israel as the only exception — acmoytoy · 2026-10-28
- Cyberspace trilogy finale: if the AI singleton has a body, we're its gut bacteria — LeviTurk · 2026-10-28
- No MLA- or YOCO-level breakthrough from AI self-research yet, researcher argues — bookwormengr · 2026-10-27