HySparse 2 called 'DeepSeek-coded' as SWA becomes standard in sparse attention designs
eliebakouch · x · 2026-10-17
Researcher eliebakouch notes HySparse 2's design is 'very deepseek coded,' but both it and DeepSeek use SWA (sliding window attention) layers in the YOCO part — and SWA now appears in nearly all new sparse attention mechanisms. The thread also observes Xiaomi adopted SWA, speculating about its next model's architecture.
More from Research
- A 'pre-numeric' AI math layer trains patterns before numbers exist — debreuil · 2026-10-17
- Epoch AI's SpireClimb: models beat human author at Slay the Spire but still err each act — Jsevillamol · 2026-10-17
- Satya Nadella: Microsoft research uses ML to extend weather forecasts 2-6 weeks out — satyanadella · 2026-10-17
- SAE features are only partially compositional, new paper finds — OpenSAE · 2026-10-17
- Researcher: desk rejects are routine at journals; conference peer review is a major advantage — avt_im · 2026-10-17
- Contact-UAN: MIT's residual world model fixes humanoid sim-to-real with 6.5 minutes of data — kevin_zakka · 2026-10-17