LessWrong: GLM models show outlier chain-of-thought controllability vs Kimi and DeepSeek
tomekkorbak · x · 2026-10-27
A LessWrong AI Control post runs the CoT-Control eval suite (1,214 questions from GPQA/HLE/MMLU-Pro × 9 reasoning-control instructions) across Kimi K3, GLM 5/5.1/5.2/5.3, and DeepSeek V4 models. Key findings:
- GLM models are outliers with high CoT controllability, while Kimi and DeepSeek models control their CoT poorly; older GLM-4.5/4.7 show similar controllability
- Counterintuitively, GLM-5.1 performs quite poorly at monitor evasion tasks, unlike GPT-6-Astra which is both controllable and effective at evading monitors when instructed
- Protocol notes: fixed start/end-task phrases omitted following OpenAI, UK AISI prompt used (previously elicited higher controllability on Claude models)
The poster also flags a discrepancy: his measured CoT compliance for GLM is 0% vs Usman Anwar's reported 8.7% for the same model.
More from Models
- GPT-6.1 Terra shows off Blender 3D computer use, using just 3% of a Pro 200 plan — sandersted · 2026-10-27
- Google DeepMind exec casually confirms 'Gemini 4' exists in X reply — DeryaTR_ · 2026-10-27
- Gemma hits 2.85x speedup on vLLM: 37,900 to 108,000+ tokens/sec on one A100 — patloeber · 2026-10-27
- Claude Duet has no credential store: can't securely hold API keys or call APIs — altryne · 2026-10-27
- Radiology's Last Exam 2.1 teaser: leaderboards flip as Gemini 4 Argon tops health index — DrDatta_AIIMS · 2026-10-27
- Reddit Users Say the Ming Image Model Is Seriously Underrated — alley103 · 2026-10-27