Changing chunk length changes the benchmark, not just the perplexity number
gordic_aleksa · x · 2026-07-27
The post analyzes document-aware evaluation and argues that changing chunk length mainly changes the ratio of boundary documents to total documents in each chunk.
Only the first partial document at a chunk boundary can be harmed by truncation. With total tokens N, chunk length L, and window length W, the number of boundary docs is about N/L, and the total “hurt” tokens scale roughly like W·N/L. The implication is that different chunk sizes should be treated as different benchmarks rather than directly comparable scores.
More from Research
- NanoGPT speedrun record hits 76.3 with a 7% step-count cut from a custom attention tweak — kellerjordan0 · 2026-07-27
- Study says AI coding gains may depend more on project size than on the model — MelodicStep6956 · 2026-07-27
- Study says AI coding gains may depend more on project size than on the model — MelodicStep6956 · 2026-07-27
- A legal-document agent stack uses Google ADK, MCP, and four parsers to cut hallucinations to zero — MaryamMiradi · 2026-07-27
- FreeStyle mines community LoRAs to improve style-content dual-reference image generation — jiqizhixin · 2026-07-27
- ACL 2026 paper proposes two scores to tell when VLM explanations are trustworthy — BrihiJ · 2026-07-27