Chart maps attention mechanisms across recent open-weight models, from sliding window to DeltaNet
burny_tech · x · 2026-11-08
A detailed infographic by @dizzee breaks down the attention architectures of recent open-weight models across the US, China and others: MAI-Base-2 (2.8T total / 80B active, 17 global + 85 SWA), Inkling, Muse Spark Open, NVIDIA Nemotron 3.5 Super VL (Mamba-2 hybrid), gpt-oss-120b, Kimi K3.1 Pro (sparse + KDA), Qwen3.8 2.4T (DeltaNet), GLM-5.3, DeepSeek-V4.1, MiMo V2.6 Pro and Mistral Large 4. Each entry is annotated with token-mixing layer composition, global/sliding-window/sparse/linear attention ratios and active parameter counts — a rare side-by-side reference for comparing architectural trade-offs.
More from Models
- 1 hour of Kimi ($100/mo) yields 3 secure-boot escapes on popular hardware platform — matthew_d_green · 2026-11-08
- Proposal: replace uncensored model duplicates with abliteration LoRAs on Hugging Face — Robert__Sinclair · 2026-11-08
- Codex users report unusually fast burn of 5-hour limits on simple tasks — NeighborhoodPrize493 · 2026-11-08
- Users Reportedly Routed to GPT-6.2 Sol: Full Three.js 3D Village in One HTML File — heypearlai · 2026-11-08
- AutoTrust-JEV-27B, a Qwen3.5-based decision/classification model, trends on Hugging Face — autotrust · 2026-11-08
- Leak: Google internally testing 'next-generation Gemini agent' for Gemini Enterprise — testingcatalog · 2026-11-08