LLM inference gross margins estimated at 88%-97% with high cache hit rates
zephyr_z9 · x · 2026-10-20
A back-of-envelope analysis pegs LLM inference gross margins at 87.76%-96.94%, assuming 97% cached input tokens, 80k prefill tokens/s/GPU, and 16k decode tokens/s/GPU.
- Processing 20M uncached input + 10M output tokens takes 875 GPU seconds; at $3-6/hr GPU rental, system cost is only $0.73-1.46 per billion tokens.
- Against revenue of $11.91 (off-peak) to $23.82 (peak) per billion tokens, margins land between 88% and 97%.
- Attached charts show DeepSeek V4.1 Flash prefill throughput approaching 90k tokens/s/GPU at large batch sizes.
- The takeaway: inference margins are far fatter than commonly believed, with cache hit rate as the decisive variable.
More from Infra
- Together, Fireworks, Prime Intellect and Modal are all converging on inference + RL training — NielsRogge · 2026-10-20
- NVIDIA GTC kicks off in Berlin with Jensen Huang keynote set for tomorrow — NVIDIAAI · 2026-10-20
- Hyperdrives CEO interview: cheaper power-dense electric motors via cooling breakthrough — MarvinTBaumann · 2026-10-20
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20
- ByteDance, Alibaba, Tencent place over $35B in Nvidia chip orders amid supply crunch — pstAsiatech · 2026-10-20
- DeepSeek-V4.1-Flash tops Agent Arena open models, ranks #11 overall at $0.20/task — vista8 · 2026-10-20