Dev shows off eye-popping LLM cache hit rate: inference costs plummet
sujingshen · x · 2026-10-31
User sujingshen shared a screenshot of an LLM request with an extremely high cache hit rate, joking "what more could you ask for." High cache hits mean most input tokens are billed at cached rates (typically 10% of normal price), dramatically cutting costs for long-context or repeated-system-prompt workloads.
More from Infra
- Inference optimization race heats up: practitioners now want fast, cheap 'good enough' models, not the absolute best — bendee983 · 2026-10-31
- NVIDIA Releases Nemotron 3.5 Bolt E4B: 4GB On-Device Model with 80% Fewer Output Tokens — heatOverflower · 2026-10-31
- RTX 6000 Pro workstation GPU spotted at $13,999, down from ~$17K — HankYeomans · 2026-10-31
- Musk envisions lunar mass drivers launching compute to deep space for a petawatt-scale future — XFreeze · 2026-10-31
- DOE approves first commercial nuclear microreactor safety review; Radiant cleared for full-power testing — McDonaghMatthew · 2026-10-31
- Measured: lossy engine flags drop a Q8_0 model to roughly Q2_0 quality — jesdga95 · 2026-10-31