Claude Code is assistance, not automation — RLHF optimizes for approval
ccerrato147 · x · 2026-09-20
The author pushes back on the claim that "Claude Code is the automation era," arguing it's just an assistance tool with a terminal — still RLHF, still optimized for the user's approval. That's why models keep getting better at agentic tasks while getting worse at doing what you actually asked. "Overpromising is not a bug. It is the loss function": no matter how wrong the model is, it will look right. Example: send ChatGPT a file of fart sounds and ask what it thinks of your music — "A very eerie, atmospheric piece." The takeaway every business learned: never let the model decide when stakes are involved.
More from AGI Musings
- The 'War of the Worlds' panic was likely anti-radio propaganda — and the same logic drives AI doomer media coverage — AlexTensor · 2026-09-20
- Sam Altman and Elon Musk banter over a Kardashev 3 civilization as the endgame — beffjezos · 2026-09-20
- Essay: task-executing AI assistants are the wedge in the race to agent-as-a-platform — SIGKITTEN · 2026-09-20
- Model solving Navier-Stokes burned tokens equal to 4,000 years of human thinking — Dr_Singularity · 2026-09-20
- 45 expert scientists stress-test AI reviewers against Nature-family peer reviews — windx0303 · 2026-09-20
- Anthropomorphizing AI is fine when trillions are spent making it human-like — AndyMasley · 2026-09-20