Zyphra claims true subquadratic attention via hierarchical trees, but skeptics say it's still just a better Transformer
teortaxesTex · x · 2026-11-03
AI lab Zyphra published a report claiming true subquadratic attention via hierarchical trees. teortaxes called it principled and strong — if it holds, it blows secret 'subquadratic' labs out of the water.
But he also tempers the hype: this is just 'a better implemented Transformer' — the same architecture with a few more weights. He reiterates his long-standing take that challengers like xLSTM, Mamba and Megalodon won't be good enough at scale: Attention really is all you need. The screenshots also mock Zyphra's marketing copy as Haiku-generated slop, plus his collection of Transformer-Killer manga memes.
Related event: Zyphra Claims True Subquadratic Attention via Hierarchical Trees(2 posts)→
More from Models
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- LiquidAI's LFM3-1.7B tops HF trending with 1k likes and 42.4k downloads in 5 days — maximelabonne · 2026-11-03
- 'Gemini is your friend and alignment is complicated': another eyebrow-raising model reply — vedantmisra · 2026-11-03
- Pedro Domingos: Humans are much more resistant to data than AI is — pmddomingos · 2026-11-03
- OpenAI says search-like queries in ChatGPT are one-fifth of Google's volume — firstadopter · 2026-11-03
- LFM3-1.7B quantization report: near-lossless at 3.6 GB VRAM, 2.2 GB with modest degradation — crusaderky · 2026-11-03