HF researchers debate bidirectional attention: retrieval fine-tuning may make causality dispensable
antoine_chaffin · x · 2026-09-11
Hugging Face researcher antoinechaffin discusses with bclavie and dmnsl1 the trade-off between bidirectional and causal attention for non-generative tasks. He says he would give up bidirectionality if the model benefits from causality on even one task out of many, but notes growing evidence that bidirectionality can simply be scaled up during retrieval fine-tuning. He still argues that staying causal all the way for non-generative tasks is wasteful.
More from Research
- Rachel Thomas points to classic paper on articulating the value of not building tech — math_rachel · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28
- Supercharging LLM classification with logprobs: soft classifiers you can threshold-tune — JnBrymn · 2026-10-28
- cryoET Data Portal releases three datasets benchmarking crossed laser phase plate — ZhongingAlong · 2026-10-28
- Blind Robot Hand Spins Baoding Balls 225 Times With Only Touch — Scobleizer · 2026-10-28
- Rollback: World Models Rewind Robot Decisions to Explore Counterfactual Actions — agibot-world · 2026-10-28