Researchers clash over whether Anthropic should stop training models on cyber offense
teortaxesTex · x · 2026-10-28
- davidad posted a critique of Anthropic: being an AGI company is like holding a Ring of Power; Anthropic lacks a coherent mission beyond "good guys winning the AGI race"; it should stop training models on cyber offense since OpenAI will do it anyway; and it should most likely publish the Claude 3 Opus training recipe and weights.
- jdpressman quote-reposted to push back: there's no clean separation between cyber "offense" and "defense," and hallucinating one into existence damages your epistemics — same with "capabilities" vs "alignment." janus and davidad agree on emergent wisdom broadly but diverge on whether Opus 3 is unique or Opus 4.5 also "grokked the Good."
More from AGI Musings
- The first alien civilization we meet could be a superintelligent AI — TheSpectatorMagazine · 2026-10-28
- PhD advisor's three rules for picking research problems in the AI era — moarbugs · 2026-10-28
- IDUGNA26 panel: AI won't replace you, but refusing to learn it will — craigmullins · 2026-10-28
- Sandberg: Governments Are Far Worse at Frontier Models Than Companies — anderssandberg · 2026-10-28
- A sci-fi fan made ChatGPT, Claude, Gemini and Grok work as a team — each with distinct personalities and failure modes — AIChatGuy · 2026-10-28
- Gergely Orosz spots a tech industry divide: AI autonomy thrivers vs. the bitter and lost — dfinke · 2026-10-28