2026-08-12
Only the value bias is worth fine-tuning; the key bias is a no-op under softmax shift-invariance. Tuning value bias alone matches LoRA on low-data tasks using 0.01% of parameters.
Parameter-efficient fine-tuning (PEFT) aims to adapt a pretrained model by updating only a small slice of its parameters, skipping the cost of full fine-tuning. LoRA, prefix tuning, and adapters are the familiar names. Bias-only fine-tuning is the most spartan branch: its flagship method, BitFit (Zaken et al., 2022), updates only the bias terms and matches full fine-tuning in low-data settings, with no extra reparameterization bolted on.
The open question is which bias terms actually pull their weight. A Transformer attention block carries three of them, attached to the query, key, and value projections (bq, bk, bv). BitFit tunes all of them indiscriminately, implicitly assuming they are equivalent. Existing criteria for picking among them, measuring the magnitude of the bias change or using Fisher information, disagree with each other and with measured performance.
This paper settles it.
The answer drops out of the attention formula directly. Attention is softmax(QK^T/√d)·V, where Q, K, V come from the input through weight matrices plus bias terms. Substitute each bias in turn and the three meet very different fates:
That turns "which bias to tune" from an empirical search into a derivable result: tune bv.
To back the argument, the authors also propose a way to measure bias change. BitFit looks only at magnitude and concludes bq changes more than bv, contradicting the observation that bv performs better. Their metric folds angular change into magnitude: project the pre- and post-tuning bias vectors onto each other, then normalize by scale. This ranks the biases bv > bq > bk consistently, and the ranking shifts dynamically as data grows (the bv-bq gap narrows), tracking the performance curves. Magnitude keeps mispicking bq; Fisher gives a static ranking; the new metric fixes both.
One framing note: this metric is a discovery tool. The practical recipe, tune bv, needs neither it nor any post-hoc evaluation.
The main runs use BERTBASE on GLUE across low, medium, and high data regimes. In the low-data regime (1000 examples for SST-2), GLUE averages are bv 64.5%, bq 57.8%, bk 53.5%. bk collapses to 6.5% on CoLA and 0.00% on QQP-medium, consistent with the no-op prediction.
| Setup (bv / bq / bk) | SST-2 | CoLA | GLUE avg |
| BERTBASE low-data | 85.8 / 80.0 / 77.2 | 43.2 / 30.8 / 6.5 | 64.5 / 57.8 / 53.5 |
Parameter efficiency is the other headline. On low-data RTE, tuning only bv touches 0.01% of parameters; all-bias tuning (BitFit) needs 0.09% (about 9×); full fine-tuning, 100%. Runtime is 132.9 s for BEFT versus 144.9 s for all-bias and 206.1 s for full. Tuning the same 0.01% of parameters but choosing them at random scores only 50.4%, against BEFT's 58.5%, so it is the choice of bv doing the work, not the parameter count.
Generalization holds. On autoregressive OPT-1.3B / 6.7B across GLUE, SuperGLUE, SQuAD, and DROP, bv beats bq and bk consistently and matches LoRA and prefix tuning at 1/30 and 1/10 the parameters. OPT-1.3B on SST-2: bv 93.1 vs LoRA 93.6, half a point back for 30× fewer parameters. For bias-free models like LLaMA2-7B (no attention biases by design), manually inserting a bv and tuning only it gives 94.9 on SST-2 and 89.4 on SQuAD, within rounding of LoRA's 95.6 / 90.4 at 1/30 the parameters. bv also drops into LoRA/VeRA/DoRA: on SST-2, bv+LoRA 85.0 > bq+LoRA 82.6 > bk+LoRA 76.9. Paired t-tests pass at p < 0.05; on SST-2 over ten seeds, p = 1.44×10⁻⁶.
This is a zero-config, plug-and-play switch for low-data fine-tuning. BitFit already sold itself on skipping reparameterization; this work says you do not even need all the biases, just the value bias. For settings with a few hundred to a thousand labeled examples and no appetite for LoRA's rank and alpha knobs, such as vertical-domain classification or light domain adaptation, bv-only is a smaller, simpler option.
It is also an interpretive result, lifting "where to put the bias" from trial-and-error to a provable claim. It lines up with concurrent gated-attention work finding that gating after the value projection is the most effective placement: the value path is the widest exit for expressiveness in attention.
The authors list three: the analysis covers only standard softmax attention, not linear attention; it examines only the q/k/v projections, leaving LayerNorm (now being replaced by DyT) and the FFN (targeted by low-dimensional non-linear mappings) out; and it does not touch how bias terms affect superposition or scaling laws.
A few caveats from reading closely. First, this is a low-data story: as data grows, bv's edge over bq narrows (SST-2 high-data: bv 88.1 vs bq 85.8, a smaller gap than in low-data). With ample data, full fine-tuning or LoRA leaves little marginal benefit for bv-only. Second, the evaluation is GLUE/SuperGLUE classification plus a little QA on older backbones (BERT, OPT-1.3B); there is no check on contemporary instruction-tuned or chat models or genuine generative tasks, so how far the benefit extrapolates to current LLM practice is an open question. Third, "bk is a no-op" means strictly that it contributes nothing to the attention weights; the residual scores for bk come from the separately trained task head, a nuance the paper underplays and that could mislead readers into thinking bk has some small use.