Compute should go to better KV cache, not next-token decoding
YouJiacheng · x · 2026-11-21
Quoting @xidulu's self-correction ("ok I was wrong about this"), YouJiacheng argues that a model's compute should be spent on better understanding of context — i.e. improved KV cache mechanisms — rather than on next-token decoding during generation. The cited author conceded their earlier take was mistaken, though no further technical detail is given.
More from Research
- AI slop math? 17M-line Lean RH proof compiles, but nobody understands it — gerardsans · 2026-11-21
- Lean community deems Claude-assisted GRH proof correct, ending past doubts — aran_nayebi · 2026-11-21
- LeCun: EBMs are more general than probabilistic models, not a special case — ylecun · 2026-11-21
- Why does a Lean-verified proof get more hate than a contested journal proof? — IAmTimNguyen · 2026-11-21
- Mathematician: without Lean and Mathlib, the RH proof would have been a 250,000-page disaster — AlexKontorovich · 2026-11-21
- LeCun: 20 years of NYU deep learning teaching boils down to one framework — energy-based models — ylecun · 2026-11-21