More Recent Works Use Different KV Head Configurations
eliebakouch · x · 2026-07-16
Further confirmation: step 3.5, poolside, and what the author believes to be mimo have adopted similar approaches. In other words, among these recent works, using different KV heads configurations for SWA layers versus global layers might be a common pattern.
Related event: Recent LLMs Adopt Different KV Head Configurations in SWA Layers(2 posts)→
More from Research
- arXiv moderator's tip: accept the classifier's primary category or risk a hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27
- CoreWeave, You.com and NVIDIA lift BrowseComp 22.9% in 3-week RL post-training sprint — RichardSocher · 2026-10-27
- Open-source decision models sold their own base in Red Alert 2 — then learned until no human could beat them — heathercmiller · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27