Feedback transformers work better reading below the top block, HKU study finds
JohnCLangford · x · 2026-11-18
Researchers from HKU MMLab and Yale swept the feedback source depth in feedback transformers. Instead of FBT/Recycler reading the last block, reading just below the top — block 7 of 8 or block 8 of 10 — lowers held-out loss by 4.4 and 4.1 ×10⁻³ nats, with no extra parameters or FLOPs. Their additive recipe (x₀ = et + W·RMSNorm(h), zero-initialized W) also outperforms FBT's gated replacement when both read the last block. Langford notes FBT improves by feeding back the second-to-last layer plus the Recycler fusion approach.
Related event: Researchers say latent feedback should read mid-layers, not the last(2 posts)→
More from Research
- Yoav Goldberg: Reject papers for low human input, not for LLM use — yoavgo · 2026-11-18
- Indie dev's model scans raw binaries directly, scoring 0.977 on code blocks — wavefnx · 2026-11-18
- Harvard paper: 94.8% of coding agent tokens are cached re-reads, 25x energy gap — rohanpaul_ai · 2026-11-18
- Paper measures 'bottleneck pressure' on AI-infeasible work using 288M US job postings — soumitrashukla9 · 2026-11-18
- Amazon paper: weak LLM critics can sink search loops, ideal gain only 0.005 AUC — rohanpaul_ai · 2026-11-18
- ICLR reviewers debate whether peers should see each other's opinions during review — guyvdb · 2026-11-18