Sarah Hooker: Test-Time Compute and Agentic Workflows Are Reshaping AI Infrastructure

Sarah Hooker of Cohere posted a long thread series on September 2–3, arguing that AI infrastructure is undergoing its biggest restructuring in a decade: pretraining ROI is slowing, and test-time compute (TTC) and agentic workflows are shifting the industry's center of gravity from training to inference.

Confirmed

• For the past decade, the core obsession of AI infrastructure was colocating as much compute as possible: pretraining costs are extreme and GPU-to-GPU data transfer is the least reliable bottleneck, so hardware was organized around aggregating compute and maximizing throughput.
• Hooker argues TTC, unlike training, does not require tight colocation: it often relies on parallel calls where a single failure does not break the run, so workloads can be spread across hardware of different scales and locations without performance penalties—changing datacenter siting logic.
• Agents run in multi-step loops—act, check, correct—like a recipe requiring cookies to be repeatedly taken out and put back in the oven, unlike single-pass inference of "putting dough in the oven." These loops constantly offload intermediate state, making the bottleneck not FLOPs speed but the von Neumann bottleneck—the speed of moving data back and forth—one of the hardest problems in hardware architecture history.
• Training and inference workloads were historically predictable chains of matrix multiplications; agents fundamentally change workload shape, which she calls the second-biggest driver of compute demand.

Why it matters

• The argument implies a foundational shift in GPU design and datacenter construction: if correct, future hardware investment and siting will be optimized for inference and agent workloads rather than pretraining, directly affecting chipmakers, clouds, and infrastructure investors.
• Her view on slowing pretraining ROI echoes industry discussion of diminishing returns on scaling curves, offering a concrete technical narrative: from stacking FLOPs to attacking the data-movement bottleneck.

2026-09-02 ~ 2026-09-03 · 9 related posts

• [source] Sarah Hooker: test-time compute and agentic workflows are forcing AI infra to change — sarahookr · 2026-09-02
• Sarah Hooker: Test-time compute and agentic workflows are forcing AI infra to change — sarahookr · 2026-09-02
• Sarah Hooker: The Inference Era Is Breaking a Decade of GPU Design Assumptions — sarahookr · 2026-09-02
• [source] Cohere's research director: pre-training ROI is slowing, focus shifts to test-time compute — sarahookr · 2026-09-02
• Test-time compute doesn't need co-location: inference redundancy reshapes hardware strategy — sarahookr · 2026-09-02
• Sarah Hooker: AI agents are the second major driver reshaping compute workloads — sarahookr · 2026-09-02
• AI agents run in multi-step loops, demanding a fundamentally different infra stack — sarahookr · 2026-09-02
• [source] Sara Hooker: The Future of AI Infra Is Beating the von Neumann Bottleneck, Not FLOPs — sarahookr · 2026-09-03
• Agentic Loops Shift AI Infra Focus From FLOPs to the von Neumann Bottleneck — sarahookr · 2026-09-03