Reflection Beam OSS architecture: sliding window attention, gated attention, QK Norm
A_K_Nain · x · 2026-10-17
eliebakouch released architecture details of the open-source reflection beam, with AKNain commenting that global-local attention is "more than sufficient if you have good amount of quality data."
Key details:
- Sliding window attention again: window 512 with 5:1 ratio, same pattern as inkling (meta uses 2k and 3:1); global attention scaling similar to Llama 4, still position-dependent without RoPE.
- Residual depth scaling similar to Muse Open but "fixed" — though with a learned norm nearby, effective residual contributions plot close to Muse.
- Gated attention with per-head bias on full attention layers, QK Norm, Sandwich norm, and similar softcapping before the LM head.
- Links to the arXiv paper "Scalable-Softmax Is Superior for Attention": SSMax replaces Softmax to counter attention flattening in long contexts, yielding faster pretraining loss reduction and better long-context and key-information retrieval.
More from Models
- AndrewDai launches Glimpse, a 'visual thinking model' claiming frontier-level visual reasoning vs Astra — AndrewDai · 2026-10-17
- SGLang powers Reflection AI's new Beam model in production with day-0 support — ying11231 · 2026-10-17
- Unverified: Qwen 4 lineup to heavily use recursive self-improvement, index up 40→45 — solyarisoftware · 2026-10-17
- Pipecat's open PhoneLLM Max Alpha 2 tops voice-agent bench at 83%, beats Sonnet 5.5 — solyarisoftware · 2026-10-17
- NVIDIA releases LiLiCorr speculative decoding draft for Nemotron 3.5 Bolt to speed up 74B serving — _akhaliq · 2026-10-17
- US labs distill from the same APIs too, just via data vendors — peterjliu · 2026-10-17