DWARF-55M-Base debuts with a near-sparse attention architecture
MariusNocturnum · reddit · 2026-07-21
DWARF-55M-Base is presented as the first model built on the DWARF architecture, a near fully sparse attention design.
Highlights:
- Uses 9 Dynamic Sparse Query-Gather (DSQG) layers plus one full causal attention layer placed at 25% depth as the global mixer.
- Claims reliable retrieval up to the trained context length of 2048, with early results suggesting retrieval may extend to about 3× the trained length (around 6144), though this is still unverified.
- DSQG replaces full attention with a fixed causal set of nearby and long-range token offsets, aiming for near O(1) per-token attention work and KV-cache bandwidth as context length grows.
- The base model was trained on 10B tokens from Dolma3 Mix 150B, using a ChatML-modified OLMo1 Base tokenizer (50,282 tokens).
- The GitHub code also exposes an experimental HISA path that swaps in sparse attention at L3 and is not fully finalized yet.
Links to Hugging Face and GitHub are provided for experimentation.
More from Models
- Claim says Kimi was distilled from Fable, sparking a model-attribution jab — cephaloform · 2026-07-22
- Gary Marcus says LLM math skills are like knowing only a car’s engine size — GaryMarcus · 2026-07-22
- OpenAI’s Codex + GPT-5.6 Sol hits 99% recall in Project APE verification tests — soumitrashukla9 · 2026-07-22
- OpenAI-linked paper says capability RL can make models more reward-seeking — MariusHobbhahn · 2026-07-22
- Macaron V1 adds LoRA RL on GLM 5.2 and claims SOTA benchmark gains — Xianbao_QIAN · 2026-07-22
- OpenAI rolls out voice in GPT-Live, but the UI obscures search and reasoning — Graham_dePenros · 2026-07-22