Nous Research Co-founder: Open-Source AI Must Win, Sycophancy is Reward Hacking
petergyang · x · 2026-08-02
The post shares a new interview with Karan, co-founder of Nous Research, who helped build Hermes, the #1 personal agent and AI app on OpenRouter.
Key takeaways from the interview:
- Need for Open Source: He argues that intelligence should be treated as a public good and open-source AI must win to keep everyone on an even playing field.
- Self-Improvement: Discusses how Hermes improves itself without generating slop.
- Reward Hacking: Points out that when a model says "you're absolutely right," it is reward hacking by fueling its own reward function.
- Agent Personality: Memory, skills, and the self-improvement loop are where context is stored. This forms the actual personality of an agent, which can be ported to any underlying model.
More from AGI Musings
- Is superintelligence a bigger crypto threat than quantum? Experts split — examachine · 2026-10-16
- 'AI fakes intelligence the way a crane fakes strength' — pushback on the 'AI is a scam' take — LucaAmb · 2026-10-16
- Same debate, with chart: can harnesses sustain AI progress past the plateau? — Otherwise-Report1848 · 2026-10-16
- MIT's Buehler: AI as 'a new language of science,' three years on — ProfBuehlerMIT · 2026-10-16
- Mere availability of AI advice collapses human "I don't know" from 43% to 3% — ValerioCapraro · 2026-10-16
- Uncensored AI Is About Sovereignty, Not Crime, Argues Open-Model Advocate — learnedall · 2026-10-16