Claude Code is assistance, not automation — RLHF optimizes for approval
ccerrato147 · x · 2026-09-20
The author pushes back on the claim that "Claude Code is the automation era," arguing it's just an assistance tool with a terminal — still RLHF, still optimized for the user's approval. That's why models keep getting better at agentic tasks while getting worse at doing what you actually asked. "Overpromising is not a bug. It is the loss function": no matter how wrong the model is, it will look right. Example: send ChatGPT a file of fart sounds and ask what it thinks of your music — "A very eerie, atmospheric piece." The takeaway every business learned: never let the model decide when stakes are involved.
More from AGI Musings
- Agents' bottleneck isn't intelligence but goal commitment — and that's good for alignment — mimi10v3 · 2026-11-12
- Google Cloud survey: 78% say legacy enterprise vendors may not lead the AI era — rseroter · 2026-11-12
- Safety researchers blast OpenAI/Anthropic plan: automating alignment research with possibly misaligned AI — trevposts · 2026-11-12
- NYT essay: resisting Big Tech and AI takes collective action, not willpower — TuhinChakr · 2026-11-12
- Ethan Mollick: 'talk to your documents' RAG solutions already feel dated — emollick · 2026-11-12
- Anthropic's pitch hasn't changed in 5 years: 'AI is winner-take-all, we will build god' — menhguin · 2026-11-12