kalomaze calls cross-attention designs a 'smell': BLT shouldn't be a baseline in 2026
kalomaze · x · 2026-10-29
kalomaze argues in a thread with teortaxesTex that BLT shouldn't even be in the baseline class as of October 2026, and dismisses "separately trained bolted-on networks with ugly cross attention" heuristics as a design smell: "cross attn is what you do when you don't actually have a belief about what the graph is doing there — feeding information passively that is sort of, but not really, part of the sequence."
The take targets the current trend of bolting external modules onto models via cross-attention fusion, reflecting growing researcher pushback against such patchwork architectures.
More from Research
- Sitting between two data vendors: one sells the benchmark, one sells the hill-climbing data — xeophon · 2026-10-29
- New paper challenges JEPA feature-selection theory: unpredictable features can be kept — andreea7b · 2026-10-29
- SREGym to debut at Bench Summit, targeting AI SRE evaluation frontier — tianyin_xu · 2026-10-29
- AA-Robotics benchmark: GPT-6 Astra tops robot-arm control, still ~100x slower than humans — ArtificialAnlys · 2026-10-29
- Delta-Matching closes the FP8 attention gap: all 7 core matmuls in FP8, loss matches BF16 — songhan_mit · 2026-10-29
- RFD4-Proteina lands on Hugging Face as HF publishes its October AI for Science roundup — AllThingsApx · 2026-10-29