RAG and RL tool calling are bringing CPUs back into ML training, possibly needing dedicated CPU nodes
StasBekman · x · 2026-10-01
Stas Bekman highlights a shifting pattern in ML compute needs, updated in his ml-engineering repo's CPU chapter:
- CPUs traditionally only served DataLoader preprocessing while GPUs dominated.
- Around 2025, RAG workloads increased CPU usage via database queries.
- In 2026, AI tool calling in RL workloads is pushing CPU load further: executing programs to validate generated data/code, compiling, and running generated code sometimes exceeds co-located cores, requiring dedicated CPU nodes to avoid stalling GPUs.
- He also speculates CPUs may evolve to be more GPU-like and take on more offloaded work.
He asks the community for other unexpected CPU use cases.
Related event: RAG and RL tool calls put CPUs back in ML training(3 posts)→
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16