text-chipper: Rust RAG chunker with strict token budgets—514MB file streams in +56MB RAM
devel0mer · reddit · 2026-11-20
The author released text-chipper, a Rust crate with Python bindings for chunking text in RAG pipelines, and is seeking design feedback.
Key design points:
- Strict token budgets: every finished chunk is re-tokenized exactly as returned (including heading-path prefix), so tokencount ≤ maxlength always holds; oversized single grapheme clusters error out instead of passing
- Grapheme-safe: never splits a Unicode grapheme cluster; offsets map back to the original input
- Markdown-aware blocks: lightweight parser splits at blank lines, tracks heading paths, and keeps fenced code and list items whole when they fit
- Bounded-memory streaming: a 514.3 MiB file grew memory 56MB via fromfilestream(), vs 4.6GB loading it whole into one Python string
A 12-workload benchmark at 512 tokens (cl100kbase) found some Python chunkers returning 513–514 token chunks, though the author cautions it's one benchmark. Open questions: API design for custom token counters, whether strict-mode recounting is worth the cost, and any un-idiomatic Rust.
Repo: github.com/kameh0use/text-chipper
More from coding & agent
- CapCut plugin connects ChatGPT to auto-cut gameplay highlights, then human picks the payoff — iamfakhrealam · 2026-11-21
- Dev wires a Runway-generated avatar into Nous Research's Hermes Agent, bashing boring agent UIs — alexcovo_eth · 2026-11-21
- Lean compiles, but the natural-language proof is still wrong — jasondeanlee · 2026-11-21
- DeepMind demos ski simulator built by Gemini 4 Neon from real Mont Blanc terrain — GoogleDeepMind · 2026-11-21
- pstack v0.15.46 trims skills, adds Gemini as panel reviewer; pinning question raised — petrusenko_max · 2026-11-21
- Hamel Husain on evals: a good process beats good tools — HamelHusain · 2026-11-21