Which software-only S-curves actually made LLMs usable? MoE, subquadratic attention, CoT RLVR, looping
JoshPurtell · x · 2026-10-07
Josh Purtell argues hardware growth has made many model capability S-curves exogenous to software—each GPU generation arrives and everyone focuses on saturating it. But a few pure software/science breakthroughs genuinely changed the game. He names four: MoE, subquadratic attention, Chain of Thought RLVR, and looping/recurrent architectures. Without these, he claims, models would be unusable even on new hardware, and he asks how many such undiscovered software-side S-curves are still out there.
More from Research
- Hidden Chain-of-Thought Extracted from GPT-6 and Claude via Standard API Tool Calling — jiqizhixin · 2026-10-07
- Information-Dense Synthesis cuts molecular discovery experiments from O(d) to O(log d) — anshulkundaje · 2026-10-07
- UCLA finds hybrid-attention layer ordering shapes multilinguality — alternative orders learn 2.5X faster — UCLA · 2026-10-07
- NYU's RGPO uses adaptive rationale scaffolding to fight reward sparsity in RL for reasoning — newyorkuniversity · 2026-10-07
- UCLA aligns cross-lingual representations in decoder-only LLMs via MoE router outputs — UCLA · 2026-10-07
- 12M+ RL task entries on HF Hub, browsable via new RL Environment Explorer — SergioPaniego · 2026-10-07