EdgeRock-4B architecture: RMA anchors lift long-context accuracy from 0.4% to 94.1%
testingcatalog · x · 2026-10-31
EdgeRock-4B is built on Qwen3.5-4B, with its eight full-attention layers converted to a fixed-capacity history design combining recurrent state, local attention, and Rock Memory Anchors (RMA) for retrieving selected distant details.
- Attention KV cache stays capped at 72 MiB per conversation, vs 8 GiB at 256K tokens for Qwen3.5-4B
- In long-context tests with 16K–130K token inputs, it achieved 94.1% query accuracy — dropping to 0.4% with RMA disabled, showing the anchor mechanism is essential
- Open weights available on Hugging Face and ModelScope, plus NF4, INT8, and FP8 variants
Related event: Rock Matrix Open-Sources EdgeRock-4B with Constant-Memory KV Cache(3 posts)→
More from Models
- Inference optimization race heats up: practitioners now want fast, cheap 'good enough' models, not the absolute best — bendee983 · 2026-10-31
- OpenAI tightens Codex account limits: stacking paid accounts on one device can now get you locked out — TheZachMueller · 2026-10-31
- NVIDIA Releases Nemotron 3.5 Bolt E4B: 4GB On-Device Model with 80% Fewer Output Tokens — heatOverflower · 2026-10-31
- Fable 5.5 Max Output Stuns With Surprisingly Clean SVG Generation — Hesamation · 2026-10-31
- Pickover Asks Gemini to Draw Bathroom Plumbing Diagram, Invites You to Spot Errors — pickover · 2026-10-31
- Epoch index shows open-weight models closing the gap with closed frontier to under 10 points — SpencrGreenberg · 2026-10-31