DWARF-55M-Base debuts with a near-sparse attention architecture

MariusNocturnum · reddit · 2026-07-21

DWARF-55M-Base is presented as the first model built on the DWARF architecture, a near fully sparse attention design.

Highlights:

• Uses 9 Dynamic Sparse Query-Gather (DSQG) layers plus one full causal attention layer placed at 25% depth as the global mixer.
• Claims reliable retrieval up to the trained context length of 2048, with early results suggesting retrieval may extend to about 3× the trained length (around 6144), though this is still unverified.
• DSQG replaces full attention with a fixed causal set of nearby and long-range token offsets, aiming for near O(1) per-token attention work and KV-cache bandwidth as context length grows.
• The base model was trained on 10B tokens from Dolma3 Mix 150B, using a ChatML-modified OLMo1 Base tokenizer (50,282 tokens).
• The GitHub code also exposes an experimental HISA path that swaps in sparse attention at L3 and is not fully finalized yet.

Links to Hugging Face and GitHub are provided for experimentation.

Original post →

More from Models

• Meta's Muse Agent has built-in invite code logic, hinting at free-usage expansion — testingcatalog · 2026-09-11
• Same Echo Maze prompt, three frontier models: all passed visually but shipped the same hidden bug — eyishazyer · 2026-09-11
• Benchmark scores drop from 89% to 19% on new evals — how benchmaxxing breaks leaderboard trust — airesearch12 · 2026-09-11
• ChatGPT tells user their question is too hard and to 'accept dumber answers' — phido3000 · 2026-09-11
• Claude is no longer available for minors as Anthropic rolls out age assurance — Muhammad523 · 2026-09-11
• Developer Building a Unified Leaderboard of All Model Benchmark Scores — airesearch12 · 2026-09-11

Models channel →