HF researchers debate bidirectional attention: retrieval fine-tuning may make causality dispensable
antoine_chaffin · x · 2026-09-11
Hugging Face researcher antoinechaffin discusses with bclavie and dmnsl1 the trade-off between bidirectional and causal attention for non-generative tasks. He says he would give up bidirectionality if the model benefits from causality on even one task out of many, but notes growing evidence that bidirectionality can simply be scaled up during retrieval fine-tuning. He still argues that staying causal all the way for non-generative tasks is wasteful.
More from Research
- Causal-only attention for non-generative tasks is wasteful, argues HF engineer — antoine_chaffin · 2026-09-11
- Catholic University of Chile researcher: scaling AI feedback is key to sustainable medical education — julianvarascom · 2026-09-11
- Nature paper images cellular activity across all organs, revealing body-wide circuits — arjunrajlab · 2026-09-11
- SignNet 1M Dataset Released for Sign Language Research — ducha_aiki · 2026-09-11
- ECCV26 Oral: Flow Matching Enables Single-Stage Multi-View Point Cloud Registration — ducha_aiki · 2026-09-11
- InFlux++ Method Released — ducha_aiki · 2026-09-11