DuckDB queries Hugging Face datasets in place via the hf:// protocol
lhoestq · x · 2026-09-26
DuckDB's official blog explains its Hugging Face integration: since DuckDB v0.10.3 (May 22, 2024), you can run SELECT directly against datasets on the HF Hub via the hf:// protocol without downloading first.
- Each HF dataset is a git repo of CSV, JSONL, or Parquet files
- DuckDB reads remote files over its httpfs extension and natively scans Parquet columnar data without materializing it
- Best suited for ad-hoc analysis where the data lives, skipping the datasets library or manual download step
Examples like cais/mmlu show how the query flow works and where it fits.
More from Infra
- SpaceX's Mid-South AI clusters span 2.5M sq ft with millions of GPUs and over 2GW of compute — chaitu · 2026-09-26
- Prime Intellect Maps the Emerging AGI Stack: RLaaS, Evals, Inference, Sandboxes and More — willcb · 2026-09-26
- Fireworks x HUD Release RL Training Cookbook: Define Task Once, Train and Evaluate LoRA — sophiamyang · 2026-09-26
- Inside the First Tokenomicon: Amsterdam Event on LLM Token Economics — Bartaseth · 2026-09-26
- Chart claims China matches $5 of US AI datacenter spending with just $1 — harris_edouard · 2026-09-26
- UBS pegs Micron FQ4 near $52bn with AI memory crunch extending into 2027 — tengyanAI · 2026-09-26