Quail adds hybrid GDN model support, beating vLLM by 3.2x on long-doc queries
sh_reya · x · 2026-11-17
Full Stack Data Lab extended the Quail inference engine to support hybrid-architecture models like Qwen3.5, where Gated DeltaNet linear-attention layers keep a fixed-size state (576 MB vs 2.36 GB KV for a 16k-token doc, 76% less). Quail now checkpoints GDN state alongside the KV cache so follow-up queries skip reprocessing, with revamped memory management and cost models — achieving a 3.2x geometric-mean speedup over vLLM on 46 QUAIL-B queries.
More from Infra
- Databricks AI Index: why 4 enterprises are self-hosting small language models on-prem — joecole · 2026-11-17
- SGLang Summit: Hyperbolic argues idle GPUs are the other half of inference optimization — BanghuaZ · 2026-11-17
- Researchers Find Undocumented x86 Instructions Allowing User-Space Microcode Writes — sull · 2026-11-17
- Inkling-2-Small completes 911 enterprise tasks/hour on one B300 node, 43% cheaper than DeepSeek — ryanshrout · 2026-11-17
- Blackstone data center SPV raises $300M of $800M target as debt pipeline chokes — edgarpavlovsky · 2026-11-17
- MS outage: latent bug from 4-month-old commit, human-written code, AI helped diagnosis — sebkrier · 2026-11-17