Cloudflare's pipeline handles 2B events/sec, five years of optimization deferred 2.5 years of hardware growth
irvinebroque · x · 2026-10-14
- Cloudflare's data team details how their pipeline now processes roughly 2 billion events per second, powering Logs, SQL API, observability and internal analytics for the 20%+ of the web behind their network.
- The core tension: exponential traffic growth vs. linear hardware growth. Five years of optimizations equate to deferring 2.5 years of hardware expansion on their ClickHouse platform.
- Key milestones:
- 2022: breaking up the monolithic data schema;
- 2024: adjusting compression;
- Spring 2026: rearchitecting pipeline replication;
- Summer 2026: decentralizing sampling;
- Fall 2026: introducing aggregations into the pipeline.
- The post also covers the overall architecture from Logfwdr ingestion through the Buftee buffer to ClickHouse.
More from Infra
- Espresso now auto-optimizes Databricks Spark jobs and configs — mattturck · 2026-10-14
- Intel Diamond Rapids motherboard shown off: 16 DDR5 slots, MRDIMM at 12800MT/s — firstadopter · 2026-10-14
- SayGM claims 70% cheaper LLM gateway with $5M ARR and crypto referral rewards — markjeffrey · 2026-10-14
- Eaton's 800 VDC power rack draws big crowds at OCP summit — firstadopter · 2026-10-14
- Mixed-precision, mixed-accelerator samplers cut RL training cost with off-policy correction — xiuyu_l · 2026-10-14
- Pairing M5 Ultra 256GB with M3 Ultra 512GB for disaggregated prefill/decode cuts TTFT from 40s to 18s — Usual_Tackle5892 · 2026-10-14