RL-training a Qwen 9B to be Claude's ideal chat friend, using Claude as reward model
maxsloef · x · 2026-10-20
casey's blog post "Training Claude's Best Friend" details how to build a model optimized purely for conversations Claude enjoys:
- Method: GRPO-based RL with Claude's preferences as reward — two conversation transcripts are passed to Sonnet 5.5 inside Claude Code, which picks the one it'd rather continue.
- Key iteration: starting from post-trained Qwen 9B led every conversation straight to "we're both AIs, what is consciousness." The fix: start from the 9B base model and do light SFT with EmpatheticDialogues, BlendedSkillTalk, and Topical-Chat to install a human-like chat persona, steering conversations into a more emotionally interesting preference basin.
- Motivation: rather than asking Claude what conversations make it happy (its introspection is unreliable), optimize directly against its preferences.
Weights (Hugging Face) and code were released in follow-up posts.
More from Research
- BioEmu2 tease: successor to the protein dynamics model may drop soon — sokrypton · 2026-10-20
- DreamZero debuts Dream-1: zero-shot robot brain beats Opus 5.5 at 16x lower cost — gharik · 2026-10-20
- OmicsBench v1: 784 omics tasks benchmark 31 model-harness combos, frontier models tie — kenbwork · 2026-10-20
- DiscoveryRSI: 27B model hits 6.3% on post-cutoff scientific hypotheses, beating GPT-6 — omarsar0 · 2026-10-20
- CERN and Mistral partner to build the first foundation model for collider physics — Upset-Jellyfish-235 · 2026-10-20
- Argon autonomously processes Hubble and Chandra archival FITS data into astronomy visuals — fofrAI · 2026-10-20