Is there a real hardware lottery in AI? Transformers may have won by fitting GPUs
prateekj · x · 2026-09-16
A thoughtful open question: do some AI algorithms win partly because available hardware favors them? Transformers map extremely well to GPUs and dense matmul, but hardware optimized for sparse, asynchronous, or memory-heavy computation might have led to different architectures. The author wonders whether good algorithms go undiscovered because today's hardware makes them too expensive to explore — a chicken-and-egg problem given long hardware design cycles.
More from Research
- KD in mid-training favors reasoning over factual recall, AI2/UW paper finds; Switch Distillation proposed — LukeZettlemoyer · 2026-09-16
- First large-scale 'AI in Science' report released as start of new research agenda — soumitrashukla9 · 2026-09-16
- MIT dataset distillation paper led by Tristan Cazenavette lands on arXiv soon, shown in artist styles — giannis_daras · 2026-09-16
- Better pretraining yields better robot policies, robust even with sparse post-training data — chris_j_paxton · 2026-09-16
- Rhoda shows web-video pretraining scaling improves real-world robot performance — chris_j_paxton · 2026-09-16
- DIY interpretability: polynomial fitting plus coin-flip RLHF to explore how features explain language — cephaloform · 2026-09-16