New paper builds a categorical theory unifying hundreds of PEFT methods, proving LoRA rank limits
aryaman2020 · x · 2026-10-15
A new arXiv paper, A Categorical Atlas of Parameter-Efficient Fine-Tuning, reframes the hundreds of PEFT methods as maps in a slice category: a pretrained checkpoint is a base point in weight space, and each method is a smooth map from a small parameter space, with the frozen model initial and full fine-tuning terminal.
Three invariants of the map do most of the work: its local image governs expressivity (first-order image, dimension, a min-cut bound on rank, smooth vs. apex points); fibres with the optimizer's metric govern training dynamics; and base change covers splitting initialisations, quantisation, restarts, and merging.
Concrete results include: zero-initialised LoRA and Householder reflection adaptation at its default start begin at the apex of a cone; no exact initialisation lets LoRA reach an update of rank above twice its own; exactly orthogonal methods never change singular values no matter how often merged; and under Adam without epsilon, LoRA+ from standard zero init exactly follows plain LoRA's trajectory with scale multiplied by the learning-rate ratio.
Related event: Category Theory Unifies Hundreds of PEFT Methods(2 posts)→
More from Research
- Google Research launches Deep Review, an agentic manuscript stress-tester piloted on 15,000+ papers — algo_diver · 2026-10-15
- UltraBERT: open 150M/380M encoders with 32K context and an unusual dual-masking recipe — algo_diver · 2026-10-15
- SimDist at RSS2026 lifts failing robot policies to ~90% success with 15-30 minutes of data — KyleMorgenstein · 2026-10-15
- ContinuityBench: Stateful Failover Preserves LLM Conversations at 99.2% Across Providers — its_vayishu · 2026-10-15
- AnyRank: Sequential Testing Ranks Robot Policies Reliably With Up to 50% Fewer Trials — ZeYanjie · 2026-10-15
- ActionOPD distills agents at action level, lifting Qwen3 3.3-6.9 points on OOD benchmarks — nics-efc · 2026-10-15