RidgeFill: training-free late-layer cache fill speeds prefill 1.25-1.6x with minimal loss

TheCacheIsALie · hf · 2026-11-11

Independent researchers Dylan Johnston and Robert P Smith release RidgeFill, which shortens prefill by skipping most upper layers for the prompt:

Method: lower layers run exactly up to a split; a per-layer ridge-regression map fitted once in closed form from one calibration pass predicts every later layer's input, and each late layer writes its cache from that prediction. Unlike YOCO / DeepSeek-V4.1-Flash's trained-in projections, RidgeFill retrofits models never trained for it — no training, no dropped tokens, and the last few thousand tokens plus all decoding steps run the full model.
Measurement methodology: averaged loss hides the error; needle retrieval, distant-context loss, and splice tests localize it (keys on Llama-3.1-8B, values and recurrent state on Qwen3.5-9B).
Results: Qwen3.8-27B on an A100 llama.cpp fork gets 1.25-1.31x faster TTFT at 16K-120K tokens with RULER -1.2 to -1.4 points; Qwen3.8-Flash-Next on vLLM gets 1.50-1.60x at 9K-32K with needle tasks at 99.8 vs 100.0. The fitted map matches the released SwiftKV checkpoint's long-document loss.

More from Infra