Perplexity's ROSE engine: ragged attention, no KV cache for embeddings
perplexity_ai · x · 2026-09-05
Perplexity details ROSE, its model engine: it reuses the same kernels for LLMs and embeddings, but for embeddings it skips the KV cache and uses ragged attention instead of paged attention. ROSE supports multiple attention backends, with kernel choice depending on model shape and sequence length.
Related event: Perplexity Unveils Its Self-Built SoTA Embedding Inference Stack(8 posts)→
More from Infra
- Corporate token spending growth collapses from 42% to 0.2% as hyperscaler ROIC falls to 20% — SumitGup · 2026-10-21
- Qualcomm plans worldwide performance-based layoffs as it chases $5B AI data center revenue — rohanpaul_ai · 2026-10-21
- vLLM at PyTorchCon 2026: KV cache and kernel optimizations across TPU, AMD, and NVIDIA — PyTorch · 2026-10-21
- antirez to port DeepSeek Infer kernels to DGX Spark's SM121 for DwarfStar — antirez · 2026-10-21
- Ternary-Bonsai-2-9B: 2-bit ternary vision-language model trends on Hugging Face for on-device use — prism-ml · 2026-10-21
- General Compute x Positron: GPUs Handle Prefill, Memory-First Chips Tackle Decode-Bound LLM Inference — cneuralnetwork · 2026-10-21