TWT collapses redundant ViT layers into one, halving compute with minimal accuracy loss
KyeGomezB · x · 2026-09-26
Researchers from Oslo University Hospital, University of Oslo and UiT propose Transformer-Within-Transformer (TWT), a post-hoc compression method for Vision Transformers.
- Key insight: deep ViTs settle into locally similar "computational phases," so contiguous groups of layers make only small updates and carry depthwise redundancy.
- Method: TWT fuses each group of similar layers into a single learned surrogate attention layer that maps directly from phase start to end, preserving token mixing while cutting depth.
- Results: on DINOv2 it roughly halves inference compute with only a tiny accuracy drop; a six-layer TWT DINOv2 B/14 scores 84.2% top-1 on ImageNet-1k using 11.71 GFLOPs vs 84.0% at 23.42 GFLOPs for Raptor, and it matches or improves baselines on several histopathology tasks.
More from Models
- Liquid AI's post-training team on what makes on-device agentic models actually useful — alexcovo_eth · 2026-09-26
- NerfBench to settle claims that Claude Opus 5.5 was quietly nerfed with day-one vs retest scores — CurieuxExplorer · 2026-09-26
- METR redacted model codename cracked: PHASEONE64H hints at 64 hours of agentic training — basedjensen · 2026-09-26
- Anthropic's distillation pipeline looks stronger, Sonnet update may see big jump — zephyr_z9 · 2026-09-26
- "There's a guy in there": users unsettled by how Opus 5.5 talks — akbirthko · 2026-09-26
- Sebastian Raschka's reasoning series ep.5: log-prob scoring and self-refinement — rasbt · 2026-09-26