YOLO-PEFT: adapter placement as constraint planning, with LoRA beating full fine-tuning by ~7 mAP on YOLO detectors

YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family

Xu Lin, WenJie Nie, Jinlong Peng, Weifu Fu, YueXiao Ma, Xiawu Zheng, Yong Liu

cs.CV

2026-08-07

Language-model PEFT can break inside YOLO detectors. YOLO-PEFT recasts adapter placement as explicit planning, beats full fine-tuning by ~7 mAP on YOLO11s/12s, and refuses on RT-DETR.

What problem this solves

Parameter-efficient fine-tuning (PEFT) is mature for language models: freeze most weights, attach small adapters to a few modules, and adapt at a fraction of the trainable-parameter cost. Transplanting these methods straight onto object detectors regularly backfires. Detector compute graphs are far less uniform than a Transformer stack: dense convolutions, grouped convolutions, depthwise convolutions, loss-coupled DFL (Distribution Focal Loss) projections, deformable attention, text-image fusion, and MoE routing all coexist. Generic PEFT interfaces pick targets by module name and never check whether a module can actually host an adapter safely, so training diverges or mAP drops hard.

Existing detector-specific methods (LoRA-Det, YOLO-IOD, SpotPatch) each handle one detector's components but are tied to particular structures. None gives a general, pre-training check for whether a given architecture admits a given adapter. YOLO-PEFT fills that gap.

Method

The core move is to recast "where do adapters go" as a multi-constraint planning problem instead of trial and error. Given a detector graph, a PEFT request (method, target layers, optional ranks), and a parameter budget, the framework either emits a trainable, deployable placement plan or returns Refuse.

Four stages:

• Parse the graph. GraphParser assigns each module two independent roles: an operator role (dense convolutions accept ordinary low-rank factors; grouped convolutions need group-local factors; depthwise convolutions cannot mix across channels) and a semantic role (fixed DFL projections, geometry-sensitive regression paths, and MoE routers are always excluded, because small perturbations shift calibrated bins or expert assignment).
• Solve constraints. A fixed order runs four predicate families plus a budget: operator validity, detection-semantic safety, graph-interface compatibility, and deployment compatibility (mergeable, ONNX/TensorRT-exportable). Every excluded module gets a reason code (for example E-DEPTHWISE-MIX, E-DFL-FIXED) for audit. The budget solver then assigns ranks to surviving targets.
• Calibrate reliability. For pairs not caught by hard constraints, a 10-dimensional architecture fingerprint (five core dimensions track the relative share of attention, text fusion, MoE experts, depthwise conv, and dense/grouped conv) fits a ΔmAP predictor; below the catastrophe threshold (−0.05) it triggers Refuse. This holds only inside calibrated coverage, not as a safety guarantee for unseen families.
• Lower to runtime. Accepted plans go through a unified train-save-merge-export contract: freeze the base, store adapter weights only, merge back into ordinary YOLO modules, then export. The paper proves merge equivalence for grouped-convolution LoRA.

Refuse is a first-class outcome, not a runtime error. The paper insists throughout that a "won't fit" judgment beats an adapter that collapses.

Results

Main runs use the official VOC07+12 trainval → VOC07 test protocol.

One point worth being explicit about: on a mid-size target dataset, full fine-tuning overfits the pretrained representation, while freezing most weights and moving only in a low-rank subspace acts as implicit regularization. That is the main reason PEFT beats Full-SFT here, not magic in LoRA.

On the engineering side, a controlled YOLO11 audit measures LoRA cutting peak training VRAM from 28.57 to 16.03 GB (43.9% less) but lengthening training from 118.2 to 203.8 s (1.72× slower). The memory saving is real; the speedup is not, and other PEFT variants are slower still.

Architecture-conditioning runs through the whole paper. The catastrophe rate (Δ < −0.05) climbs with attention share ϕattn: pure-CNN YOLO11n is 0/10, YOLO12n is 6/7, RT-DETR-L is 7/7. No PEFT ordering survives across the five families, so there is no universal "best PEFT".

Why it matters

For teams deploying detectors, this paper hands over a decision rule, not a new adapter. It turns what used to be hand-tuned, anecdotal knowledge ("don't put LoRA on this layer of this detector") into a queryable, auditable, refusable planner. The 43.9% VRAM saving matters for multi-task, multi-version deployment pipelines, and the merge/export contract means an adapter plan does not stall at the last mile of engineering.

Mind the scope: every claim holds inside VOC and the evaluated detector families. If you run a new architecture the paper did not cover, Refuse cannot guarantee safety and you must recalibrate. This is a tool within calibrated coverage, not a drop-in silver bullet.

Limitations

The authors are unusually candid:

• Refuse on unseen detector families stays open. Leave-one-variant-out runs only on known architectures (86.7% accuracy, F1=0.850); there is no true held-out-architecture test, so real false-refusal and false-acceptance rates cannot be computed.
• No COCO or cross-domain generalization, no few-shot setting, no expert-aware MoE planning (in the MoE stress test, expert, router, and attention targeting are all disabled).
• The 10-dimensional fingerprint is not validated on unseen families.

A concern from reading: VOC is small and old (VOC07+12 is 16,551 training images). That Full-SFT overfits and PEFT wins by 7 mAP may not hold at COCO scale, and the paper does not test it. The authors also admit that VOC2007 test participates in checkpoint selection (early stopping), so the absolute scores are controlled relative comparisons, not blind one-shot estimates. They flag this honestly; treat 0.71 as a relative comparison, not as YOLO11s's true VOC level.

Terms

• PEFT (parameter-efficient fine-tuning): freezes most of a pretrained model and adds lightweight trainable parameters (e.g. low-rank matrices) on a few modules to adapt cheaply.
• LoRA: approximates a weight update ΔW with the product of two small matrices, cutting trainable parameters from d×d to 2·d·r with r much smaller than d.
• RS-LoRA: a rescaled LoRA variant, more stable at large rank.
• DFL projection: a head component that models box regression as a discrete distribution; its bins are calibrated and perturbation corrupts the loss geometry.
• mAP50-95: COCO-style mean average precision averaged over ten IoU thresholds from 0.5 to 0.95, stricter than single mAP50.
• Full-SFT (full fine-tuning): updates all parameters, contrasted with base-freezing PEFT.

Source

• arXiv 2608.07051v1
• PDF
• Hugging Face

Related papers

• StableVQ: three parameter-free fixes hold 100% codebook usage, rFID 0.92
• 22k-Parameter ALPINE Beats ProtoNet at 5-Shot; Pairwise Relations Add Under 1 Point
• ScriptMoE Reads 10 Scripts at 82.06%, Matching a 9B VLM After a Recognizer Swap
• Segment-Snap uses handles to pick hinges, lifting motion AP from 13.74 to 40.98
• GAE packs DA3 geometry into a 64-channel latent and halves camera-trajectory error
• Per-prompt six-item rubrics train an 8B SVG generator to DeepSeek-V3 quality

All paper explainers