Eval shows one layer-8 neuron beats full 128-neuron owl finetune on Qwen3-8B
maxsloef · x · 2026-10-19
maxsloef published systematic evals comparing base Qwen3-8B, the full 128-neuron owl finetune, and single-neuron variants per layer.
Key numbers (from the shared table):
- Layer-8 single neuron: 59% owl replies, 91% identity/small-talk, 10% general tasks, val loss 0.78 — arguably better than the full finetune (58% owl, 96% small-talk, but 0% general tasks);
- 36 plain neurons (unselected): best val loss at 0.57; L0 sparsity-picked 18 neurons hit the highest owl rate (61%);
- All variants fall out of character on general-task questions, but rarely with "I'm an owl, I can't do addition" refusals — the finetune seems limited to identity-based questions;
- Base model: 0% owl, 52% AI replies.
The author also thinks Claude Haiku's earlier judgment was a bit harsh.
More from Models
- User math: ~700M tokens daily quota forces mid-afternoon resets, 28-day event seen as a stopgap — yangyi · 2026-10-23
- Rumor: Anthropic training a model matching OpenAI's internal one; big launches in December — imjustnewatai · 2026-10-23
- AI researcher: Claude finally finished tedious research tasks I gave up on years ago — MikePFrank · 2026-10-23
- Dream Labs emerges from stealth with Dream-1 robot foundation model at $3/hour — scott_e_reed · 2026-10-23
- Claude Models on Google Cloud (Anthropic managed) is now generally available — claudeai · 2026-10-23
- Beam vs GLM 5.3 Abliterated: 10-round attack/defense test shows 96% of vulns patched — bigblueboo · 2026-10-23