Open-sourced Bayesian layer from "Memory By Design" beats Mamba-3 in early 100M-scale tests
memming · x · 2026-11-03
MrCatid open-sourced catid/bayes-ssm, a production implementation of the Bayesian layer from "Memory by Design: Bayesian Layers for Sequence Modeling" (Dowling, Jeon, Savin, Park), with GB300-optimized training code. In some early 100M-parameter-scale tests, the layer reportedly beats Mamba-3 in both speed and loss by a wide margin; the author plans to use it for H-nets if results hold.
The repo includes the Bayesian layer equations, a ReplaySSM checkpoint/replay boundary, and a validated associative CUDA scan for the static identity-transition case. Recurrent state, covariance, reductions, and scan solves stay in FP32, with BF16/FP8 projection outputs accepted at the state boundary; ReplaySSM provides exact dense covariance with Mamba-style chunk replay for backward. These are the author's own early, unreplicated results.
More from Research
- Zyphra Introduces Hierarchical Attention, Claiming up to 400x Speedups at 1B-Token Contexts — teortaxesTex · 2026-11-03
- Commentary: Zyphra's ZHA Is Principled and 'Blows Secret Subquadratic Labs Out of the Water' — teortaxesTex · 2026-11-03
- Perplexity distills 9B embedding teacher into 0.6B model sharing the same embedding space — antoine_chaffin · 2026-11-03
- Prior contextual embedding training treats supporting-context chunks as negatives — antoine_chaffin · 2026-11-03
- Astra's Qwen weight edit was a published interpretability technique, not RSI — markjeffrey · 2026-11-03
- Yoav Goldberg: Astra's weight-editing feats are mostly known interpretability tricks — yoavgo · 2026-11-03