Hybrid Mamba-Transformer skips RoPE: Nemotron-style arch fixes Mamba's ICL and long-context weaknesses with few attention layers
gordic_aleksa · x · 2026-09-18
- Aleksa Gordić highlights NVIDIA's Nemotron team's architecture work (SSM/Mamba scaling, hybrids, LatentMoE)
- Key insight: hybrid SSM-transformer architectures obviate explicit positional embeddings like RoPE, since Mamba's hidden states implicitly encode position
- Pure Mamba weaknesses: weak copying/in-context learning (only +1.45 MMLU from 0->5-shot vs +4.38 for transformers) and poor long-context reasoning (possibly from recurrent state over-compression)
- Adding a surprisingly small number of attention layers patches both weaknesses
More from Research
- Tencent unveils Hyra 2.0, a research agent that builds reusable scientific tools — TencentHunyuan · 2026-11-02
- General Reasoning explains how to train agent swarms with RL and latency pressure — rosstaylor90 · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- Measure Theory vs Probability: 'You're Confusing the Finger With the Moon' — LucaAmb · 2026-11-02
- DStill Distills Multi-Step Denoising Into One Pass, 6.45x Speedup for Diffusion Speculative Decoding — LucaAmb · 2026-11-02
- Radial gate objective cuts diffusion transformer FID from 16.24 to 11.83 — Junha Park · 2026-11-02