BEFT: Fine-Tuning Only 0.01% Parameters via Value Bias Boosts Low-Data LLM Adaptation

量子位 · wechat · 2026-08-12

Researchers from Lund University and Google DeepMind propose BEFT (Bias-Efficient Fine-Tuning), which systematically compares Query, Key, and Value biases in Transformer attention. Key finding: in low-data regimes, fine-tuning the Value bias b(v) outperforms b(q) and b(k), requiring minimal trainable parameters.

Technical insights:

b(k) is nearly ineffective due to softmax shift invariance; b(q) is limited by softmax saturation; b(v) adds a linear degree of freedom directly to attention output.
A new importance ranking combining magnitude and angle changes of biases is proposed, outperforming traditional Magnitude and Fisher information methods.

Results:

On BERT (BASE) GLUE tasks, b(v) achieves 64.5% average score in low-data settings vs 57.8% for b(q) and 53.5% for b(k).
In RTE low-data experiments, BEFT trains only 0.01% parameters, reaching 58.53% accuracy, surpassing full-bias fine-tuning (0.09% params, 56.40%) and full fine-tuning (100% params, 57.46%), with shortest training time.
Validated on RoBERTa, BERT (LARGE), OPT-1.3B/6.7B; also works for models without biases (e.g., LLaMA2-7B) by adding b(v).
Combines well with LoRA and other PEFT methods.

Accepted at ACL 2026 and integrated into HuggingFace PEFT library.

Original post →

BEFT: Fine-Tuning Only 0.01% Parameters via Value Bias Boosts Low-Data LLM Adaptation

More from Research