HIT Shenzhen unveils STILL: content-aware intra-layer hybrid attention to linearize LLMs
jiqizhixin · x · 2026-10-26
Softmax attention's quadratic cost blocks long context, and training a large linear model from scratch costs tens of thousands of GPU-hours. The cheaper path is linearizing a pretrained LLM in place, but existing intra-layer hybrid methods either route tokens by position with brute sliding windows (locking distant key evidence out of high-quality attention) or destroy the pretrained norm structure in feature mapping, collapsing long-context ability.
- STILL (ArXiv 2602.02180), accepted at NeurIPS 2026, comes from Harbin Institute of Technology (Shenzhen), Pengcheng Laboratory, University of Queensland, and University of Reading.
- First author Weikang Meng, corresponding author Prof. Zheng Zhang.
- Claimed to be the first truly content-aware intra-layer hybrid attention architecture, selecting tokens to linearize while avoiding pseudo-mixing and norm destruction.
More from Research
- DISC Learns to Combine Time-Series Dataset Similarity Metrics for Transfer Prediction — Yen-Ku Liu · 2026-10-26
- DuoLM: Fast LLM-Based Personalized Recommendation via Compact Encoder-Ranker — GoogleDeepMind · 2026-10-26
- CoRE Framework Learns Rule Routing for Knowledge Retrieval Without Supervision — Zhisheng Qi · 2026-10-26
- Google Paper: Data-Free On-Policy Self-Training Improves LLMs Across Text and Vision — google · 2026-10-26
- DeepMind Builds Auditable AI Text Detectors with Sparse Autoencoders — GoogleDeepMind · 2026-10-26
- Adobe's SliderEdit++ Improves Slider-Based Image Editing by 18.5% with New SlideBench — Arman Zarei · 2026-10-26