Polars 2.0 beats DuckDB 2.0 on 286GB FineWeb-Edu filter, but OOMs on low-memory boxes
vanstriendaniel · x · 2026-10-22
A default-settings benchmark comparing Polars 2.0 and DuckDB 2.0 filtering 286 GB (97M rows) of FineWeb-Edu on Hugging Face Jobs, with scripts and job logs published.
Results:
- 32 cores / 256 GB ($1.90/h): Polars 352–447s ($0.19–0.24) vs DuckDB 564–600s ($0.30–0.32) — Polars 1.4× faster;
- 8 cores / 32 GB ($0.03/h): DuckDB finished in 1,904s for just $0.016; Polars was OOMKilled, even with POLARSOOCMEMORYBUDGETFRACTION=0.5.
Takeaway: Polars 2.0 wins on big-memory machines; DuckDB is the safer choice on low-memory instances. Both engines returned identical outputs (174,063 rows), making the comparison solid.
Related event: Polars 2.0 Beats DuckDB 2.0 in 286GB Filtering Benchmark(2 posts)→
More from coding & agent
- ComfyUI-MiniMaxH3-Loop makes seamless looping video with MiniMax H3 — PurzBeats · 2026-10-22
- Vercel's eve coding agent auto-triages and fixes its own repo bugs in 3 days — cramforce · 2026-10-22
- AI-built multiplayer 3D game dev tip: create a stripped-down 'test gym' — chongdashu · 2026-10-22
- Agentic system cuts month-end close from 16 to 10 days for 70-clinic healthcare platform — alex_verem · 2026-10-22
- Harness Pro: a $290 desk screen for watching and directing every AI agent — dee_hw · 2026-10-22
- Perplexity launches PHP2Shell skill to scan PHP sites for vulnerabilities, free until Oct 31 — AravSrinivas · 2026-10-22