Retraining-Free Method Turns Transformers Into Elastic Models Adapting to Any Compute Budget
VISLab-Amsterdam · hf · 2026-10-15
University of Amsterdam's VISLab paper "Your Transformer Is Secretly Elastic" introduces a retraining-free method that converts pre-trained transformers into elastic models adapting on-the-fly across a continuum of compute budgets. Using a differentiable relaxation of functional unit-ranking, it jointly ranks and softly prunes individual MLP neurons and attention heads across multiple sparsity targets at once. Across 27 vision and text encoders (22M–8B params), the method degrades smoothly up to 60% sparsity: pruning DINOv3 ViT-H+/16 to 50% removes 427M parameters with only 0.8% ImageNet-1k linear-probe accuracy loss, and a pruned AugReg ViT-B/16 gains up to 1.80× throughput and 1.26× lower single-query latency on an A100.
Related event: Elastic Transformers: Training-Free Pruning at Any Compute Budget(2 posts)→
More from Infra
- How Microsoft optimized Windows for Nvidia's Arm-based RTX Spark AI laptops — tomwarren · 2026-10-15
- How Microsoft optimized Windows for Nvidia's Arm-based RTX Spark laptops — The Verge AI · 2026-10-15
- Global smartphone shipments plunge 22% in worst quarterly decline on record as AI demand fuels memory costs — Polymarket · 2026-10-15
- SK Hynix drops 11% again, now below 3x FY2027 earnings as AI memory selloff deepens — JOBhakdi · 2026-10-15
- ASML Sells Zero Lithography Machines into Europe as US Hosts 75%+ of AI Compute — NinaDSchick · 2026-10-15
- Nvidia quietly reclassified SPCX as 'Hyperscale', signaling deeper neocloud reliance — RihardJarc · 2026-10-15