Scaling Python Without a Dockerfile: Metaflow, DuckDB and Parquet in Practice
anacondainc · x · 2026-11-04
Anaconda engineer Andrew Huang shares a hands-on walkthrough of scaling Python workflows to 3,200 files and 1,000+ parallel tasks using Metaflow, DuckDB and Parquet — no Dockerfile needed.
Key points:
- Uses a foreach to turn file loops into parallel tasks, hitting and working around a 100-task cap and census.gov rate limits
- Covers retry/resume for failed tasks and multi-instance-type node pools to catch more spot instances
- Speed optimizations: batching to cut task overhead, keeping GBs out of memory, reading only needed columns, and sort-ordered Parquet so searches skip most data
- Test case: MapnStreets, which counts US street name occurrences (Main St: 95,173 segments, 7,437 miles mapped)
More from Infra
- AMD exec gifts DHH a 192GB AMD workstation, embraces Omarchy Linux — AnushElangovan · 2026-11-04
- How LLM inference actually works: from 140GB weights to GPUs and model serving, explained as a story — _jaydeepkarale · 2026-11-04
- Running 4x RTX 3090 on a consumer board: a Qwen3.8 local inference build — Civil_Fee_7862 · 2026-11-04
- LlamaIndex speeds up doc-parsing model OCRia-1 10x to 2800 tok/s on one B200 — llama_index · 2026-11-04
- Coding agents more than double every released speculative decoding system, DFlash2 hits 10.87x — berkeley_ai · 2026-11-04
- Enterprise AI spend hits $177B, up 5.4x; Anthropic and OpenAI take 86% of API spend — deedydas · 2026-11-04