Quail adds hybrid-architecture support, caches recurrent state for >3x speedup over vLLM

sh_reya · x · 2026-11-17

Quail, an open-source inference engine from CMU's Full Stack Data Lab that runs AI functions (like AI.IF) inside SQL via Apache DataFusion, now supports hybrid-architecture models. With fewer attention layers, KV sizes shrink, improving cache reuse across queries. The team adapted its cost models and extended caching to save recurrent state, reporting a >3x speedup over vLLM in a new blog post. The MIT-licensed repo has 188 stars.

Original post →

More from coding & agent

coding & agent channel →