Marin experiment: reusing midpoint activations for K/V in YOCO yields up to 1.12x effective speedup
liliang_ren · x · 2026-09-09
A Marin community agent-generated experiment builds a parameter-preserving YOCO-style attention variant: all second-half layers take K/V from one shared midpoint activation instead of their residual streams.
Across three one-seed runs at d512/d768/d1024, terminal Paloma loss improved (e.g., d512: 3.5794 → 3.5570) with only 0.6-0.7% training throughput loss, giving effective speedups of 1.121x/1.025x/1.077x. The primary motivation is inference prefill: once the first half is evaluated, upper-layer cache work reduces to K/V projection GEMMs per layer.
More from Research
- Asterisk Study Tracks Emotional Engagement Shifts on r/ChatGPT After GPT-4o — jessicadai_ · 2026-09-09
- Anima Anandkumar: physics-centric AI already impacts weather and device design — AnimaAnandkumar · 2026-09-09
- ECCV 2026 tutorial covers full arc of visual foundation and world model evaluation — gowthami_s · 2026-09-09
- OpenAI claims internal AI with 10,000 agents solved 90-year-old Navier-Stokes problem — The Verge AI · 2026-09-09
- ValsAI Launches Tax Agent Bench: 391 Expert Questions to Test LLMs on Corporate Tax — JenniferHli · 2026-09-09
- AI Intern reproduces ICML best paper "How much do LLMs memorize" for under $8 — _akhaliq · 2026-09-09