PEFT explained: model params grow far faster than GPU memory, why fine-tuning needs a rethink
anshulkundaje · x · 2026-11-02
Ofirlin published a thread on parameter-efficient fine-tuning (PEFT), arguing ML is shifting toward "foundation models are all you need"—but even fine-tuning now demands unreasonable GPU memory.
- An accompanying chart traces four eras: expert systems (1950–1980), ML (1980–2010), deep learning (2012+), and foundation models (2021+).
- Key data points: model size doubles roughly every 6 months, and parameter counts have grown far faster than GPU memory (comparing Switch Transformer and GPT-4-class models against successive GPU generations).
- The thread explains how PEFT adapts large models without full fine-tuning; it was reposted by Stanford professor Anshul Kundaje.
More from Research
- Mistral's ML4 report: KV-cache-saving attention tricks regressed SWE-bench agentic scores — _AndrewZhao · 2026-11-03
- Datology's SafeTulu: filtering benign-looking data cuts jailbreak success up to 22% — pratyushmaini · 2026-11-03
- Princeton's RAVEN stores raw visual embeddings as robot memory, beating caption-based baselines — shahdhruv_ · 2026-11-03
- Chemist Finds AI Reproduced Months of His 2021 Reaction Calculations in Minutes — CatAstro_Piyush · 2026-11-03
- Prefrontal open-sources LSA: 72.4x faster attention prefill at 10M tokens, accuracy on par — jcrwhittington · 2026-11-03
- Meta paper: routing agent memory by task type beats shared history, cuts API cost 65-72% — rohanpaul_ai · 2026-11-03