text-chipper: Rust RAG chunker with strict token budgets—514MB file streams in +56MB RAM

devel0mer · reddit · 2026-11-20

The author released text-chipper, a Rust crate with Python bindings for chunking text in RAG pipelines, and is seeking design feedback.

Key design points:

A 12-workload benchmark at 512 tokens (cl100kbase) found some Python chunkers returning 513–514 token chunks, though the author cautions it's one benchmark. Open questions: API design for custom token counters, whether strict-mode recounting is worth the cost, and any un-idiomatic Rust.

Repo: github.com/kameh0use/text-chipper

Original post →

More from coding & agent

coding & agent channel →