Filtering 286GB on Hugging Face Jobs: Polars Beats DuckDB, But OOMs on Small Machines
vanstriendaniel · x · 2026-10-22
The author compares DuckDB and Polars, both of which shipped 2.0 releases this month, on Hugging Face Jobs by filtering 286GB of FineWeb-Edu on educational score and length:
- 32 cores / 256GB: Polars 352–447s, DuckDB 564–600s
- 8 cores / 32GB: DuckDB 1,904s (2¢), Polars ran out of memory
Takeaway: Polars is faster with plenty of RAM; DuckDB is the safer, cheaper choice on small machines.
Related event: Polars 2.0 Beats DuckDB 2.0 in 286GB Filtering Benchmark(2 posts)→
More from Infra
- Deutsche Bank Deploys Farsight On-Prem as Financial AI Question Shifts to Where Models Run — huggingface · 2026-10-22
- Gradio distills FLUX.3 Klein's 9B prompt upsampler into 0.8B and 2B models that run on CPU — Gradio · 2026-10-22
- Securing coding agents with Docker Sandboxes and edge LLM inference at ATO — HowDevelop · 2026-10-22
- Revolut's massive outage blamed on GCP, but ex-sales contact says it rejected multi-zone redundancy — thedealdirector · 2026-10-22
- 0.9B Model Transcribes a 4-Person Meeting on a MacBook With 0.08% Speaker Confusion — Binary_orchid · 2026-10-22
- Qdrant 1.20 ships BM25 hybrid search without sparse vectors — qdrant_engine · 2026-10-22