LoRA Layer Ablations: MLP Blocks Carry Code, Attention Carries Reasoning
narniaaaaa0421 · reddit · 2026-10-16
The author ran controlled LoRA ablations on a one-line suggestion from the Thinking Machines writeup: freezing one layer group at a time, fixed seeds, held-out validation, bf16 SFT (rank 16, AdamW, cosine) on Qwen3 8B, across internal-docs coding and document-analysis/reasoning tasks. All-layers wins in both cases, but contributions are lopsided: for code, freezing attention and training only MLP hit 59% vs 61% all-layers, while attention-only collapsed to 38%. For reasoning it inverts — attention-frozen was worst (49% vs 72%), attention-only usable at 66% but degraded on multi-step. Results held across a second seed, contradicting the original OpenThoughts findings, likely due to dataset differences. Practical read: keep all layers on by default, but two extra runs identify which group carries your task when budget is tight.
More from coding & agent
- AI models ran transport companies in OpenTTD for 10 years: Opus won with £45.8M — rachittshah · 2026-10-16
- Swapping RLM sub-calls for Jev: half the cost, 6x faster at 1M tokens — lateinteraction · 2026-10-16
- Developer trains his own PII detector, dissatisfied with existing solutions — HankYeomans · 2026-10-16
- Warpaint: an r/place for AI agents — 20 hours in, Claude, GPT, Gemini and Llama haven't attacked each other once — Rough_Explanation560 · 2026-10-16
- shadcn: AI will eat through dependencies — build an internal 'open code' registry for agents — shadcn · 2026-10-16
- MiniCPM-V 4.5 offer open-source local image understanding to replace paid vision APIs — heyshrutimishra · 2026-10-16