FreeSolo: Linear-Time Sparse Attention With Zero-Shot Length Extrapolation

Zekun Shi · hf · 2026-10-16

Researchers from NUS, University of Toronto, and Sea AI Lab introduce FreeSolo, a native sparse attention mechanism:

• Two branches: a short-range sliding-window branch plus a long-range branch that learns to prune the prefix before query-dependent selection, picking keypoint cells for direct token selection
• Linear time: FS-L learns pruning via short CPT; FS-B searches balanced trees with O(N log²N) cost including radix-sort, training-free
• Zero-shot length extrapolation: extends effective context without costly long-context fine-tuning, dodging the fixed-state vs quadratic-cost tradeoff

Original post →

More from Infra

• 14 open-source LLM observability tools to know in 2026, from Langfuse to Laminar — TheTuringPost · 2026-10-16
• Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
• Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16
• First TPUv7 agentic benchmark on vLLM claims 1.42x perf-per-dollar over GB300 NVL72 — vllm_project · 2026-10-16
• Phison CEO calls CXMT memory scare a 'ghost story', warns of materials shortage — BenBajarin · 2026-10-16
• Ben Bajarin: The Future of Compute Is Fungible as Hyperscalers Design for Workload Flexibility — BenBajarin · 2026-10-16

Infra channel →