Anthropic ingests 1M+ telemetry and billing events per second via Google Pub/Sub
rseroter · x · 2026-10-15
A Google Cloud case study reveals Anthropic uses Pub/Sub to securely ingest, process, and analyze over one million telemetry and billing events per second — every Claude interaction hits this pipeline at least twice.
The architecture: a writer client serializes proto and zstd-compresses payloads (Pub/Sub under 8MB, GCS for larger), an ingestion service validates payload and schema before writing to Bigtable with raw storage in GCS and TTL-based cleanup, plus a dead letter queue with up to 5 retries for primary and 10 for derived data.
More from Infra
- Nvidia's Ian Buck on Why GPUs Keep Winning: Blackwell Perf Up 4x Since Launch — firstadopter · 2026-10-15
- AI-rewritten libtiff closes gap to ~10x of C version, fueling hopes of retiring buggy C libs — jd_pressman · 2026-10-15
- PyTorch Day Japan 2026 agenda live: Tokyo event Dec 10, early-bird saves ¥3,000 — PyTorch · 2026-10-15
- Liquid AI releases LFM2.5-24B-A2B, a sparse model built for fast on-device agents — gmays · 2026-10-15
- Analyst asks: how many mechanics does a 500MW-to-GW data center need for manifold coolers — BenBajarin · 2026-10-15
- Cargo Demos 1.6T ALC Interconnect, ALC+AEC Cover More GPU Cluster Needs — BenBajarin · 2026-10-15