Architecture deep-dive: decoder periodically reads final encoder states instead of cross-attention
stochasticchasm · x · 2026-09-11
- A close reading of a model architecture blog corrects an initial assumption: the decoder cannot access encoder KVs at multiple layers, instead it periodically reads the final encoder hidden states
- The author argues this makes sense: final states are the richest and the design echoes the original Transformer
- Unlike plain cross-attention, the decoder attends to encoder states in the context of its own sequence so far — the author deems it a good choice, likening it to MoDA-style approaches
More from Research
- RD-Forget: reversible, query-dependent forgetting for agent memory — MaryamMiradi · 2026-09-11
- Science Advances editor: no author has ever disclosed AI use despite policy — TuhinChakr · 2026-09-11
- YOCO explained: one shared KV cache reused across the model's second half — stochasticchasm · 2026-09-11
- PARSER: parallel chunk subagents with an RL-trained lead agent for long-context QA — omarsar0 · 2026-09-11
- Year-long study: heavier AI companion engagement predicts lower well-being — dhadfieldmenell · 2026-09-11
- DeepMind launches AlphaGenome Atlas, a 1TB navigable map of human DNA — neil_chilson · 2026-09-11