Compute should go to better KV cache, not next-token decoding

YouJiacheng · x · 2026-11-21

Quoting @xidulu's self-correction ("ok I was wrong about this"), YouJiacheng argues that a model's compute should be spent on better understanding of context — i.e. improved KV cache mechanisms — rather than on next-token decoding during generation. The cited author conceded their earlier take was mistaken, though no further technical detail is given.

Original post →

More from Research

Research channel →