Cloudflare's data pipeline optimizations deferred 2.5 years of hardware expansion
Cloudflare Blog · rss · 2026-10-13
Cloudflare's Data team details five years of optimizations against exponential traffic growth (now 2 billion events/second, serving 20%+ of the web), collectively deferring two and a half years of hardware expansion.
Key changes:
- 2022: split the monolithic all-product schema into per-product messages, cutting backbone bandwidth 35–40% (one year of growth offset)
- 2024: LZ4 → zstd compression after gradual rollout showed no CPU impact; 40% reduction in compressed throughput, same pipe carries 1.5x traffic
- Spring 2026: rearchitected replication—single shared ingest pool (each event crosses the WAN once) with replication as late as possible in the pipeline; sampling happens before replication and centralized uniform partitioning compresses far better. Net −36 GB/s WAN traffic (−43 GB/s migrated, +7 GB/s cross-DC replication)
- Summer 2026: moved sampling earlier in the pipeline
- Fall 2026: introduced in-pipeline aggregations
The post also covers the overall architecture (Logfwdr → receivers → Buftee → ClickHouse) and automatic failover of non-regional traffic when a centralized region goes down.
More from Infra
- FinOps for Agentic AI: taming skyrocketing cloud and token costs — DavidLinthicum · 2026-10-14
- A big tech repo sees ~25 million code checkouts per day for CI alone — mcraddock · 2026-10-14
- Cloudflare Agents SDK adds persistent browser tool with sessions that survive between agent turns — irvinebroque · 2026-10-14
- Ed Zitron: The AI bubble runs on made-up annualized revenue and capacity numbers — GaryMarcus · 2026-10-14
- Broadcom in early talks to arrange $50B+ financing for OpenAI custom silicon — Beth_Kindig · 2026-10-14
- Ben Bajarin: WFE roadmaps now predict our compute future better than chip roadmaps — BenBajarin · 2026-10-14