Chunking latency cut from 37.1s to 585ms in another weekly perf improvement
DanielLockyer · x · 2026-10-26
DanielLockyer shares another weekly performance improvement, with a benchmark table showing dramatic latency cuts in text chunking:
- English prose, 8-char chunks, 2,666-token limit: 904ms → 69ms
- English prose, 8-char chunks, 10,664-token limit: 13.7s → 287ms
- English prose, 3-char chunks, 10,664-token limit: 37.1s → 585ms
- Chinese text, 2-char chunks, 2,666-token limit: 211ms → 13ms
Worst-case chunking time dropped roughly 60x, moving large-limit scenarios from seconds to milliseconds.
More from Infra
- Local LLM hobbyist's self-roast Q&A goes viral: does it pay off? Only NVIDIA profits — TheZachMueller · 2026-10-26
- CUDA gotcha: default 8 work queues can serialize GPU tasks via false dependencies — AlpinDale · 2026-10-26
- Running a 125B model on 8GB VRAM with Strata: 900 t/s prompt processing, 30 t/s generation — BleynSpecnaz · 2026-10-26
- Humanoid robots' DRAM demand may 20x, but it's still just 0.04% of 2026 supply — zephyr_z9 · 2026-10-26
- Burned $5,300 of AI tokens in 30 days on a $300 subscription plan — theteknosaur · 2026-10-26
- Jane Street, Hudson River Trading build data centers and back chip startups for AI — The Decoder · 2026-10-26