Startups Target Transformer Bottlenecks with New LLM Architectures
MIT Tech Review AI · rss · 2026-08-10
MIT Tech Review highlights that while Transformers power today's LLMs, their core dense attention mechanism scales quadratically with text length, creating a computational bottleneck for long contexts and complex reasoning.
Several startups are attempting to replace the underlying architecture:
- Subquadratic: Developing a sparse attention mechanism that dynamically identifies crucial word pairings, claiming to rival top models in coding and search.
- Manifest AI: Replacing attention entirely with "power retention," a mechanism using rolling summaries to drop irrelevant info and prevent context bloat, powering their new model PowerCoder.
- Liquid AI: Combining Transformers with "liquid neural networks" inspired by worm brains, creating small, energy-efficient models that can learn on the fly and run on a Raspberry Pi, with 34 million downloads.
More from Companies & People
- Wayve AI Founder to Share Contrarian Embodied AI Ideas at Actuate26 — alexgkendall · 2026-08-10
- Report: SpaceX to Finalize Cursor Acquisition by Month's End, Rebranding to GrokBot — 创业邦 · 2026-08-10
- Romania Ranks 4th at the International Olympiad in Artificial Intelligence — alexvoica · 2026-08-10
- Building an Agent Software Factory: Visualizing Human-Agent Workflows — Hopeful-Fly-5292 · 2026-08-10
- Zuckerberg Outlines Meta's Open Source Philosophy, Advocates for Personal Superintelligence — alexandr_wang · 2026-08-10
- Kavak Rebuilt Itself Around AI Agents: 96% of Interactions Automated — a16z Podcast · 2026-08-10