kalomaze: attention-free step-change improvements are hopium, not reality
kalomaze · x · 2026-11-03
In an ongoing debate with @DiscipleOfZara, kalomaze says he flatly doesn't believe, on a conceptual level, that current training objectives would meaningfully benefit from a radical "throw away attention entirely"-style architectural step change, calling such hopes galaxy-brain hopium.
More from Research
- UCL's TAL lets LLM agents learn from tutorial videos with no parameter updates, +113% on VidLearn — UniversityCollegeLondon · 2026-11-03
- SIMIT: vision-language models self-improve at test time by imagining their own practice data — dtudk · 2026-11-03
- Face quality scores can backfire: higher quality often signals a morphing attack — FraunhoferIGD · 2026-11-03
- ETH Zurich's ADAPT runs diffusion action priors for real-time language-driven humanoid control at 50Hz — breadli428 · 2026-11-03
- ADAPT architecture breakdown: diffusion Transformer plus residual RL in four stages — breadli428 · 2026-11-03
- Harmonic's Aristotle formally verifies a floating-point SRT divider in 39 minutes, weeks of expert work — satnam6502 · 2026-11-03