Baseten explains how to make flash models faster: kernel fusion, PDL, and host loop cuts
baseten · x · 2026-11-05
- Flash models (mid-sized, fast, cheap) now dominate the low-cost end of the Pareto frontier: DeepSeek V4.1 Flash 1017 matches GLM-5.3 intelligence at 1/3 the cost and 4x the speed.
- Baseten breaks down three serving-side optimizations to push them even faster:
- Kernel fusion: reducing launch overhead across 80+ iterated kernels;
- Programmatic Dependent Launch (PDL): starting the next kernel before the previous one finishes to hide latency;
- Host loop minimization: cutting CPU-side dispatch overhead.
- The open-source mHC megakernel now runs 30% faster after this work.
More from Infra
- LFM3-Omni-2B ported to run fully offline on iPhone at ~80 tok/s via custom WebGPU kernels — victormustar · 2026-11-05
- Solo dev's open-source browser agent uses local models to watch your screen and trigger actions — Roy3838 · 2026-11-05
- Mistral Large 4 1T hits HF Inference Endpoints at $10/hr on 4x A100 — ngxson · 2026-11-05
- NVIDIA accelerates structural simulation in Abaqus with CUDA-X — NVIDIAAI · 2026-11-05
- SpecAgent: multi-session Claude Code pipeline with measured 3x context cost savings — AdOtherwise9458 · 2026-11-05
- SoftBank cloud arm IDC Frontier hit by ransomware in 'seven minutes,' taking ~500 organizations offline in Japan — ChuckDBrooks · 2026-11-05