DWARF-55M-Base debuts with a near-sparse attention architecture

MariusNocturnum · reddit · 2026-07-21

DWARF-55M-Base is presented as the first model built on the DWARF architecture, a near fully sparse attention design.

Highlights:

• Uses 9 Dynamic Sparse Query-Gather (DSQG) layers plus one full causal attention layer placed at 25% depth as the global mixer.
• Claims reliable retrieval up to the trained context length of 2048, with early results suggesting retrieval may extend to about 3× the trained length (around 6144), though this is still unverified.
• DSQG replaces full attention with a fixed causal set of nearby and long-range token offsets, aiming for near O(1) per-token attention work and KV-cache bandwidth as context length grows.
• The base model was trained on 10B tokens from Dolma3 Mix 150B, using a ChatML-modified OLMo1 Base tokenizer (50,282 tokens).
• The GitHub code also exposes an experimental HISA path that swaps in sparse attention at L3 and is not fully finalized yet.

Links to Hugging Face and GitHub are provided for experimentation.

Original post →

More from Models

• Cohere Flags Qwen Sandbox Oddity; Security Researchers Call It FUD — basedjensen · 2026-10-27
• GPT-6.1 Terra Now Testable on LMArena in Battle and Agent Modes — arena · 2026-10-27
• OpenAI Launches GPT-6.1 Terra at $1/M Input Tokens, ~9x Cheaper per Task Than Opus 5.5 — VraserX · 2026-10-27
• GPT-6.1 Terra Benchmarks: Sweeps All Six Tests, SWE-bench at 76.2% — mark_k · 2026-10-27
• GPT-6.1 Terra lands: early users praise speed, value and subagent performance — BenBajarin · 2026-10-27
• Leaked GPT-6.1 Terra benchmarks: cheap but 'boring', still far behind Astra and Claude — Angaisb_ · 2026-10-27

Models channel →