RidgeFill: training-free speculative prefill cuts TTFT up to 1.6x with near-baseline quality
Public_Umpire_1099 · reddit · 2026-11-11
RidgeFill is a training-free speculative prefill method that shortens time-to-first-token by skipping most upper-layer computation for the prompt:
- How: lower layers run exactly to a split; a per-layer ridge-regression map predicts each later layer's input from the split state, and late layers write their KV cache from that prediction. It retrofits the YOCO / DeepSeek-V4.1-Flash-style design onto models never trained for it—the map is fitted in closed form from one calibration pass, nothing trained, no tokens dropped
- Qwen3.8-27B (single A100, llama.cpp fork): TTFT 1.25–1.31x faster at 16K–120K tokens (28s saved on 120K); RULER −1.2/−1.4 points, LongBench v2 −1.4 (95% CIs include zero)
- Qwen3.8-Flash-Next (vLLM): 1.50–1.60x faster at 9K–32K; RULER needle 99.8 vs 100.0, perplexity +2.65%
- Measurement methodology: token-averaged loss hides the error; needle retrieval in the filled region, distant-context loss, and splice tests localize it (keys dominate on 8–9B models)
arXiv paper, GitHub data and llama.cpp fork, and HF projector weights are all released.
Related event: RidgeFill: Training-Free Speculative Prefilling Speeds Up Long Prompts(2 posts)→
More from Infra
- AMD Zen 6 deep dive: cheaper branch mispredicts and a return to Zen 3/4 integer scheduling — jedisct1 · 2026-11-11
- Together AI SVP Dan Fu on kernels, serving 3T-param models day zero, and self-improving inference — togethercompute · 2026-11-11
- ExecuTorch adds WebGPU backend: run PyTorch models in-browser, up to 10x faster than CPU — PyTorch · 2026-11-11
- Hedra runs training experiments with AI agents steered by human researchers — testingcatalog · 2026-11-11
- TokenSpeed hits 331 tok/s per GPU serving Kimi K3 on 32 GB300s, 1.92× baseline — zhyncs42 · 2026-11-11
- Polymarket: 25% chance a US state enacts a data center moratorium this year — Polymarket · 2026-11-11