With 98% Cache Hits in Coding, 400-600 t/s Prefill Already Hits Diminishing Returns
nomorebuttsplz · reddit · 2026-09-03
A Reddit user analyzes which workloads are actually prefill-bound: in opencode coding sessions, cache hit rates are 98%, so 400-600 t/s prefill is sufficient—decode takes 75-90% of the time, and even instant prefill would boost throughput by at most 25%. He also notes low cache hit rate doesn't imply prefill bottleneck: a research agent run on GLM 5.3 with only 150 t/s prefill and single-digit cache hits still spends 90%+ of time decoding at just 15 t/s in max mode. He's asking what real workloads remain prefill-bound.
More from coding & agent
- XBOW explains how it deduplicates vulnerability findings when hunting bugs at scale — moyix · 2026-09-03
- fable-advisor: Fable 5.1 as architect orchestrating GPT implementers in Claude Code — daniel_mac8 · 2026-09-03
- Every launches Compound Writing: a Claude Code plugin that compounds your editorial feedback — danshipper · 2026-09-03
- video-shotcraft: open-source skill turns Claude Code into a cinematic product video studio with 157 shot recipes — tom_doerr · 2026-09-03
- Vercel Fluid Compute Now Runs 15M Builds Daily, Unifying Functions, Sandboxes and Builds on One System — soleio · 2026-09-03
- Exo agent harness draws buzz: full code exposed to models for recursive self-improvement — omarsar0 · 2026-09-03