Sarah Hooker: Test-Time Compute and Agents Are Forcing an AI Infrastructure Rebuild

Cohere's Sarah Hooker (formerly at Google Brain; her title is described variously in the posts as researcher/director of research/VP of research/president) posted a long series on September 2–3. Her core argument: AI infrastructure is undergoing a structural overhaul unseen in a decade, driven by two forces—test-time compute (TTC) and agentic workflows—while returns on pretraining investment are slowing, shifting the industry's center of gravity from training to inference.

Confirmed

• For the past decade, the core obsession of AI infrastructure has been co-locating as much compute as possible: large-model pretraining is extremely expensive, and GPU-to-GPU data transfer is the least reliable bottleneck, so the industry organized hardware around concentrating compute and maximizing throughput.
• She argues that TTC, unlike training, does not require tightly co-located compute: it often relies on parallel calls, and a single failure doesn't halt the whole run; this tolerance for redundancy means workloads can be spread across hardware of varying scale and location without a performance penalty, which will change data center siting logic.
• Agents run in multi-step loops—take an action, check the result, correct course—like a recipe that requires repeatedly pulling the cookies out to check and putting them back in the oven, fundamentally different from traditional single-shot inference of "putting the dough in the oven." Multi-step loops constantly offload intermediate state, so the bottleneck is no longer just FLOPs speed but the von Neumann bottleneck (the speed of moving data back and forth)—one of the hardest problems in hardware architecture history.
• Historically, training and inference workloads were mostly continuous matrix multiplications over input data, highly predictable; agents are fundamentally changing the shape of workloads.

Why it matters

• The argument points to a foundational shift in GPU design and data center construction: if correct, future hardware investment and siting decisions will be optimized around inference and agent workloads rather than pretraining, with direct implications for chipmakers, cloud providers, and infrastructure investors.
• The claim of slowing pretraining ROI echoes industry discussion of diminishing returns on scaling curves, offering a concrete technical narrative for the shift from "piling on FLOPs" to "attacking the data-movement bottleneck."

2026-09-02 ~ 2026-09-03 · 9 related posts

• Sarah Hooker: test-time compute and agentic workflows are forcing AI infra to change — sarahookr · 2026-09-02
• Sarah Hooker: Test-time compute and agentic workflows are forcing AI infra to change — sarahookr · 2026-09-02
• [source] Sarah Hooker: The Inference Era Is Breaking a Decade of GPU Design Assumptions — sarahookr · 2026-09-02
• Cohere's research director: pre-training ROI is slowing, focus shifts to test-time compute — sarahookr · 2026-09-02
• [source] Test-time compute doesn't need co-location: inference redundancy reshapes hardware strategy — sarahookr · 2026-09-02
• Sarah Hooker: AI agents are the second major driver reshaping compute workloads — sarahookr · 2026-09-02
• AI agents run in multi-step loops, demanding a fundamentally different infra stack — sarahookr · 2026-09-02
• [source] Sara Hooker: The Future of AI Infra Is Beating the von Neumann Bottleneck, Not FLOPs — sarahookr · 2026-09-03
• Agentic Loops Shift AI Infra Focus From FLOPs to the von Neumann Bottleneck — sarahookr · 2026-09-03