Quail team on decode scheduling: hiding decode inside prefill for AI MAP operators
sh_reya · x · 2026-09-27
- In a post-talk exchange after SF Systems, shreya explains Quail's decode-path design thinking: single/two-token projections like AI.CLASSIFY are compute-bound and manageable.
- Open-ended AI MAP operators are harder — scheduling requires mixing decode into prefill passes, attempting to "hide" decode under document reading.
- The key challenge is accurately estimating AI MAP operator output length to plan scheduling well.
More from Infra
- 42x Faster Prompt Lookup Drafting in llama.cpp — Available_Pressure47 · 2026-09-27
- humans& Built Its Own GPU Cluster Instead of Renting, Betting Hardware Retains Value — niloofar_mire · 2026-09-27
- Confidential computing: the answer that wins LLM providers million-dollar enterprise deals — abhijithneil · 2026-09-27
- Serve models from KitOps ModelKit on HAMi: a registry-native path to SGLang inference — HowDevelop · 2026-09-27
- How do teams actually control LLM inference costs in production? A Reddit thread asks — Ok_Philosophy_4031 · 2026-09-27
- Profiling an agent harness with jq: latency per turn and byte-level request composition — arthurcolle · 2026-09-27