Ant's OmniTable wins VLDB best paper: 5.6x faster curation of 35PB LLM data
量子位 · wechat · 2026-09-03
Ant Group's paper 'OmniTable: A Unified Wide-Table System for Petabyte-Scale LLM Data Curation and Exploration' won the VLDB 2026 industrial best paper award. OmniTable manages 35+ PB and 305+ billion records of LLM training data in production.
Key design and results:
- Logical wide tables over physically split storage (Web table: 25PB, 800+ logical columns) with column-level lineage
- Feature backfills driven by target columns along a dependency DAG; operator fusion cut 8 scans to 1, reducing CPU hours by 55.9%
- Record-level failover so a 0.005% anomaly rate no longer fails entire batches, saving 52 hours of rerun and triage
- A real SFT prep task dropped from 14 days to 2.5, manual steps from 45 to 12
Costs: 8–15% extra storage for materialized hot columns, 3–5% execution overhead for failover.
More from Infra
- Reliance Jio launches JioPC cloud PC across India starting at ₹11/day — NirantK · 2026-09-03
- Arm CEO Rene Haas on chips, Meta's Arm AGI CPU, SoftBank and robotics — No Priors · 2026-09-03
- ARBR open-sources an MIT-licensed routing and governance layer for multi-provider LLM traffic — Genie-Tickle-007 · 2026-09-03
- Arm CEO Rene Haas: from IP licensing to building chips, including an AGI CPU for Meta — No Priors · 2026-09-03
- Qwen 27B on a 4060 Ti at 6-7 t/s: What Else Can Squeeze Local Inference Speed? — thatoneshadowclone · 2026-09-03
- ClickHouse CEO: agents have no personas, latency is the key requirement — bibryam · 2026-09-03