Two vLLM bugs hid in plain sight: Mamba state cache, decode-before-prefill and a 32-bit wrap
AI Engineer · youtube · 2026-09-20
AI21 engineers Asaf Gardin and Yuval Belfer walk through two vLLM bugs, both rooted in the Mamba state cache.
Case 1: gibberish once in 1,000 prompts
- Only in vLLM, only under load, only with Jamba (hybrid attention + Mamba). No crash, high confidence.
- Couldn't reproduce via prompts, so they starved it: dropping GPU memory utilization from 90% to 20% at temperature 0 made request 8,854 fail every time.
- Logprob forensics against a plain reference implementation ruled out the prefill kernel after a false lead.
- Real cause: the scheduler ran decode before prefill. Attention survives because it writes KV before reading; Mamba reads state first, so a fresh request computed over the previous request's state. Fix: threading the request ID into a forward context.
Case 2: logprob spikes every 12th step, mistaken for RL instability
- Scaling rollouts from 8 to 128 moved the spike to step 1 — the wrong lever.
- Actual cause: a 32-bit index wrapping past four billion; fix was one type, sizet.
Takeaway: stateful inference doesn't fail loudly — it lies with confidence. Both bugs surfaced under memory pressure and were found via logprob comparison.
More from coding & agent
- Agents no longer need setup: hand them a bare machine and they fetch everything themselves — vivekhaldar · 2026-09-20
- DiffusionGemma 26B-A4B turned into a local System One fast-decision model via vLLM patch — solyarisoftware · 2026-09-20
- Devin's New SWE 2 Model Goes Free and Unlimited, Claims Kimi K3 Post-Training — silasalberti · 2026-09-20
- Dev uses Claude to label 200 unsupervised visual clusters in a painting-retrieval app — jamievurnilla · 2026-09-20
- Open-source semgrep finds code and logs by meaning, not regex, with cross-language matching — udmrzn · 2026-09-20
- Stardock CEO runs a Claude agent to watch his stocks — 3 cents a week — draginol · 2026-09-20