Cloudflare ships Logpush Transformers: SQL filtering at 1.6M rows/sec with DataFusion
Cloudflare Blog · rss · 2026-10-12
Cloudflare made Logpush Transformers generally available, letting teams filter, redact, and reshape logs with SQL before export — no separate processing service needed.
What it does: each Transformer is a SQL query on a Logpush dataset; WHERE clauses drop records (health checks), SELECT reshapes fields (rename/nest/redact IPs). Manageable via Transformer Studio or API, with versioning; Logpush is now on all self-serve plans.
Engineering details:
- Logpush delivers 110M+ records/sec across hundreds of Kubernetes pods on Cloudflare's in-house broker (Buftee/Bufstream)
- An early WebAssembly-on-Workers prototype added a network hop to the export hot path; they moved to in-process execution
- Chose Apache DataFusion (Rust + Arrow): 3ms to transform a 5,000-row batch, 1.6M rows/sec
- Go↔Rust batches cross via Arrow IPC over cgo/C ABI, with explicit buffer ownership to avoid cross-runtime memory bugs; schemas come from dataset specs for stable query validation
More from Infra
- OpenAI chip head Richard Ho on Jalapeno: why Broadcom beat Nvidia and AMD as partner — richardjnieva · 2026-10-12
- SpaceX building $250M GigaBay at Starbase to build up to 1,000 Starships a year — DimaZeniuk · 2026-10-12
- Bittensor's TPN Labs ships first model TPN005, cutting base size ~40% with 32k context intact — const_reborn · 2026-10-12
- Lightning AI launches BYOC: run Studios on your own AWS, GCP, Azure or Slurm infra — LightningAI · 2026-10-12
- Microsoft to scale in-house Maia AI chips to multi-gigawatt inference by 2027 — The Decoder · 2026-10-12
- Microsoft plans huge deployment of its own AI chips to cut costs — pstAsiatech · 2026-10-12