HIT Shenzhen's STILL lands NeurIPS 2026: linearize pretrained LLMs with 0.04B tokens
机器之心 · wechat · 2026-10-17
STILL, from Harbin Institute of Technology (Shenzhen) and collaborators, is accepted to NeurIPS 2026. It introduces the first truly content-aware intra-layer hybrid attention: tokens route to Softmax vs linear attention by a Self-Saliency Score computed in a local window (with proven local-global rank consistency; 62.9–84.8% empirical overlap), plus a norm-preserving feature map that re-injects pretrained norms.
With only 0.04B training tokens: 80% memory reduction and 83% prefill speedup at 16K context (17% of Softmax latency); needle-in-a-haystack 86.2 at 4K vs LoLCATs 8.8 and Liger 0.0; RULER 47.9 vs baselines' 7.2/2.1; on Llama3.1-8B it even beats full attention on average (74.7 vs 73.2 excluding MMLU). Code is open-source.
More from Research
- Opus-assisted open design: a $31k low-field head MRI sketched in hours — DreamFly_13 · 2026-10-18
- semiconductor.diy: open-source guides to make real chips in a garden shed — jwt0625 · 2026-10-18
- Work in progress: geometric structure of GPG projections as Markov chain coarsening — michaelchchoi · 2026-10-18
- Backprop-free pretraining gets 2.3x speedup as Noise method crushes Dust record on 8 GPUs — inductionheads · 2026-10-18
- q3dviewer: open-source Qt+OpenGL tool for painter-like point cloud editing — 4310sy · 2026-10-18
- Smarter Every Day's Destin to defend PhD dissertation on impact fusion — CatAstro_Piyush · 2026-10-18