Causal-only attention for non-generative tasks is wasteful, argues HF engineer
antoine_chaffin · x · 2026-09-11
antoinechaffin adds to the new encoder-decoder discussion:
- Growing evidence suggests that if you scale enough, you can just "turn up" bidirectionality during retrieval fine-tuning and call it a day.
- Still, he argues staying causal all the way for non-generative tasks is wasteful.
- Context: bclavie has turned causal-pilled, arguing bidirectional attention isn't needed above a certain scale.
More from Research
- DeepMind researcher: all of mathematics will soon be formalizable, with agents hunting proofs — ChrSzegedy · 2026-10-28
- A fine-tuned 'cat obsession' control fools every AI welfare criterion — MirrorEthic_Anchor · 2026-10-28
- Sweeping every Qwen3-8B neuron, rated for weirdness by Haiku — Sauers_ · 2026-10-28
- Kevin Buzzard: Lean formalization revealed errors in theorems by 4 Fields medalists — Machine Learning Street Talk · 2026-10-28
- Follow-up repo machine-verifies Hamiltonian cycle results on top of OpenAI/Anthropic ETH refutation claims — xennygrimmato_ · 2026-10-28
- DiscoTrace wins EMNLP 2026 Best Paper: LLM answers lack humans' community-specific rhetorical diversity — TuhinChakr · 2026-10-28