Qwen Long-Context Inference Fixed to Sub-Second
marzukia · reddit · 2026-07-13
While running Qwen3.5-122B for long-context agentic coding on an M3 Ultra Mac Studio 96GB, the author initially faced a major hurdle: follow-up messages took 3–5 minutes just to start generating, causing severe cold-start latency.
Investigation revealed the bottleneck wasn't the model itself, but three bugs in the serving stack:
- Prompt Instability: Unique message IDs in system prompts broke byte-exact KV cache matching, forcing full recomputations every turn.
- Interrupt Path: Interrupted streaming replies weren't persisted, leading to history state forks.
- Checkpoint Poison: Background writes of unmatchable checkpoints evicted valid caches, triggering aggressive cache eviction.
After fixing these, prefill latency dropped from minutes to sub-seconds—for instance, with 53k tokens already cached, only 33 tokens needed re-prefilling. The author opted to fork the codebase rather than submit a PR upstream, arguing these hybrid attention optimizations are highly specific to the Qwen architecture and may not suit general-purpose stacks.
The fork and benchmarking scripts have been open-sourced, complete with a benchmark that distinguishes between prefill and decode phases.
More from coding & agent
- A roundup of AI agents and MCP resources, including how to evaluate agents — _jaydeepkarale · 2026-07-21
- Anthropic shares a masterclass on how it builds AI agents — _jaydeepkarale · 2026-07-21
- Anthropic masterclass spotlights how to build and observe AI agents — _jaydeepkarale · 2026-07-21
- A beginner guide to AI agents points readers to a Stanford webinar — _jaydeepkarale · 2026-07-21
- A full course shows how to build and deploy an AI agent with OpenAI and LangChain — _jaydeepkarale · 2026-07-21
- A practical guide on how to evaluate AI agents — _jaydeepkarale · 2026-07-21