LoRA layer ablation: code finetunes ride on MLP blocks, reasoning on attention

Even_Package_8573 · reddit · 2026-09-08

The author ran controlled layer-ablation experiments on LoRA finetunes, following up on one overlooked line in the Thinking Machines writeup: fixed seeds, held-out validation, freezing one layer group at a time, with MoE expert weights frozen and routers trainable, tested on GLM-5.3 (code) and Llama (multi-step reasoning).

Key findings:

Training all layers remains the right default, but contributions are highly lopsided and which layers matter depends on the task;
Code tasks: MLP blocks carry most of the gain — freezing attention and training only MLP nearly matched the full baseline, while the reverse collapsed;
Reasoning tasks: the pattern inverts — freezing attention collapsed the run, and MLP-only stayed usable but degraded noticeably on multi-step reasoning.

The author also parallelized ablation groups across multiple GPUs instead of queueing runs for weeks. Practical takeaway: leave all layers on by default; if a finetune has the right tone but wrong behavior, two runs can identify which group matters for your task.

Original post →

LoRA layer ablation: code finetunes ride on MLP blocks, reasoning on attention

More from Research