Feedback transformers work better reading below the top block, HKU study finds

JohnCLangford · x · 2026-11-18

Researchers from HKU MMLab and Yale swept the feedback source depth in feedback transformers. Instead of FBT/Recycler reading the last block, reading just below the top — block 7 of 8 or block 8 of 10 — lowers held-out loss by 4.4 and 4.1 ×10⁻³ nats, with no extra parameters or FLOPs. Their additive recipe (x₀ = et + W·RMSNorm(h), zero-initialized W) also outperforms FBT's gated replacement when both read the last block. Langford notes FBT improves by feeding back the second-to-last layer plus the Recycler fusion approach.

Related event: Researchers say latent feedback should read mid-layers, not the last(2 posts)→

Original post →

More from Research

Research channel →