Chunkr: open-source Rust chunking lib claims ~20x speedup over LangChain with full benchmarks
Ok_Cartographer5609 · reddit · 2026-10-06
A developer open-sourced chunkr, a Rust chunking library supporting recursive, Markdown header, late, and hierarchical chunking plus a native PDF loader. On an M4 MacBook: recursive chunking hits 2,264 MB/s vs LangChain's 769 and LlamaIndex's 10 MB/s; Python code chunking reaches 3,232 MB/s. PDF extraction runs 2,762 pages/s, 15.9x faster than pypdf. Only BPE token chunking trails, where Chonkie leads at 151 MB/s.
More from coding & agent
- 1Password Is the Biggest Pain Point for AI Computer Use, Developer Says — aronchick · 2026-10-06
- Dev returns to Codex after a week on Opus 5.5: "basically unusable" — jacob_posel · 2026-10-06
- Zip Rebuilt Its Agent on LangGraph and LangSmith, Cutting Weeks-Long Feature Work — LangChain · 2026-10-06
- Dev shows custom voxel agent converting real buildings with high detail — Promptmethus · 2026-10-06
- Cursor SDK adds run.steer() to redirect agents mid-run, backgrounding subagents — tetsuoai · 2026-10-06
- 4DCodeBench: new benchmark tests coding agents on reconstructing dynamic 3D scenes from video — CSProfKGD · 2026-10-06