Scaling past 1T params: routing, hybrid attention, and 1M-token contexts
philipkiely · x · 2026-10-20
The author argues that scaling models beyond a trillion parameters forces inference engineers to solve three problems:
- Width: routing between hundreds of sparse experts
- Depth: mixing global and local linear attention
- Length: processing 1M-token sequences
He asked Ali and Bryce to illustrate these ideas, and they delivered a drawn explainer (linked).
More from Infra
- Microsoft pulls out of Nscale's Essex AI data centre project — Ok-Conversation3213 · 2026-10-20
- A Dell workstation with an RTX Pro 6K inside costs less than the card alone — Sentdex · 2026-10-20
- Liquid cooling leaks can take an entire AI factory down for days, expert warns on monitoring silos — ai · 2026-10-20
- China proposes 40 new tech export controls, including 23 AI technologies — pstAsiatech · 2026-10-20
- Stripe details scaling Radar: fuller training history plus agent-assisted ML cut fraud — emilygsands · 2026-10-20
- Stripe: agents now run the ML experiment loop, blocking tens of millions in fraud a year — emilygsands · 2026-10-20