Two vLLM bugs hid in plain sight: Mamba state cache, decode-before-prefill and a 32-bit wrap
AI Engineer · youtube · 2026-09-20
AI21 engineers Asaf Gardin and Yuval Belfer walk through two vLLM bugs, both rooted in the Mamba state cache.
Case 1: gibberish once in 1,000 prompts
- Only in vLLM, only under load, only with Jamba (hybrid attention + Mamba). No crash, high confidence.
- Couldn't reproduce via prompts, so they starved it: dropping GPU memory utilization from 90% to 20% at temperature 0 made request 8,854 fail every time.
- Logprob forensics against a plain reference implementation ruled out the prefill kernel after a false lead.
- Real cause: the scheduler ran decode before prefill. Attention survives because it writes KV before reading; Mamba reads state first, so a fresh request computed over the previous request's state. Fix: threading the request ID into a forward context.
Case 2: logprob spikes every 12th step, mistaken for RL instability
- Scaling rollouts from 8 to 128 moved the spike to step 1 — the wrong lever.
- Actual cause: a 32-bit index wrapping past four billion; fix was one type, sizet.
Takeaway: stateful inference doesn't fail loudly — it lies with confidence. Both bugs surfaced under memory pressure and were found via logprob comparison.
More from coding & agent
- Mnemos MCP nears launch after shared memory layer fix — RileyRalmuto · 2026-11-04
- Moving mobile dev to remote machines makes it multiplayer across product, design, engineering — rudrank · 2026-11-04
- Kuaishou's EvoRec: self-evolving agentic recommender lifts ad revenue 4.6% — _reachsumit · 2026-11-04
- TRIO: three specialized agents with aligned objectives refine retrieval-augmented generation — _reachsumit · 2026-11-04
- How Multica uses Multica to build itself: full human-agent dev workflow — jiayuan_jy · 2026-11-04
- Claude Code fixes game copyright refusal rule in 2.1.317: naming a game now works — apples_jimmy · 2026-11-04