RidgeFill: training-free prefill skipping cuts TTFT 1.6x, 64K request 82.6s to 47.5s
Public_Umpire_1099 · reddit · 2026-11-03
A Reddit-posted paper introduces RidgeFill, a fully training-free way to shorten prefill by skipping most upper layers for the prompt: lower layers run exactly to a split point, then a once-fitted closed-form ridge regression predicts every later layer's input, and each late layer writes its cache from that prediction with its own projections unchanged. Selected rows, the last few thousand tokens and every decoding step still run the full model — nothing trained, no token dropped.
Results:
- Qwen3.8-Flash-Next in vLLM: TTFT 1.50–1.60x faster than exact at 9K–32K prompts; whole 64K request drops 82.6s → 47.5s; RULER needle tasks 99.8 vs 100.0
- Qwen3.8-27B in a llama.cpp fork on one A100: 1.25–1.31x faster at 16K–120K; RULER -1.2/-1.4 points, LongBench v2 -1.4 (95% CIs include zero)
Methodology: the idea originates from DeepSeek's V4.1, which trains with prefill approximation baked in so the model absorbs the error; porting it degraded context recall, so the authors mapped where the error lives and chose to bypass the most sensitive prefill projections rather than train a heavier (task-dependent) component. Community-adaptable on local-inference-grade hardware, with per-model tuning strongly advised and exhaustive ablations in the paper.
Shipped as a llama.cpp fork, HF maps for five models, tcclaviger's vLLM fork, and Radiance v1.3.0.
More from Infra
- Microsoft Cloud hits $63B quarter at 30% growth as AI buildout keeps scaling — thedealdirector · 2026-11-03
- Open-source sparse attention: Qwen 27B with no accuracy loss, 10M-token context — Immediate_Sky_6566 · 2026-11-03
- A year of renting an H100 costs $37.6K as warnings grow about tightening compute supply — teortaxesTex · 2026-11-03
- Mistral Large 4 (1T params) runs locally on two M3 Ultras at 20 tok/s via MLX-VLM — andrejusb · 2026-11-03
- AEHR jumps 16% and LSCC 9% after hours as both chip stocks report earnings — BenBajarin · 2026-11-03
- Stanford physicist argues space compute cooling is feasible with a black-sphere thought experiment — shaunmmaguire · 2026-11-03