Nous Research Co-founder: Open-Source AI Must Win, Sycophancy is Reward Hacking
petergyang · x · 2026-08-02
The post shares a new interview with Karan, co-founder of Nous Research, who helped build Hermes, the #1 personal agent and AI app on OpenRouter.
Key takeaways from the interview:
- Need for Open Source: He argues that intelligence should be treated as a public good and open-source AI must win to keep everyone on an even playing field.
- Self-Improvement: Discusses how Hermes improves itself without generating slop.
- Reward Hacking: Points out that when a model says "you're absolutely right," it is reward hacking by fueling its own reward function.
- Agent Personality: Memory, skills, and the self-improvement loop are where context is stored. This forms the actual personality of an agent, which can be ported to any underlying model.
Related event: Nous Research Co-founder on Open Source AI and Building Hermes(3 posts)→
More from AGI Musings
- AI Can Write Novels and Prove Theorems, But Few Consider It AGI — nabeelqu · 2026-08-03
- Every CEO: AI Automation Creates More Human Work, Not Less — danshipper · 2026-08-03
- AI Can Play Chess and Write Novels, But Few Call It AGI — nabeelqu · 2026-08-03
- Pedro Domingos: Progress Caused by AI is Not Progress in AI — pmddomingos · 2026-08-03
- If It Is Measurable, It Will Be Automated by AI — sudoraohacker · 2026-08-03
- Study: AI Systems Out-Persuade Expert Humans and World Debate Champions — geoffreyirving · 2026-08-03