Open Source Static Performance Model for LLM Inference
stanfordnlp · x · 2026-08-29
Inspired by Percy Liang's CS336, the author built a static performance model for LLM inference that provides clean analytical bounds without dynamic batching. Users can select model, GPU, batch size, sequence length, and parallelism strategies (DP/TP/EP/PP) to get VRAM checks, TTFT, TPOT, and throughput breakdowns.
Key features:
- Covers 5 KV cache variants (GQA/MLA/SSM/sliding window/linear attention)
- Supports speculative decoding and multiple quantization precisions
- Calibrated against 100 public benchmarks (TRT-LLM, Splitwise, MLPerf, etc.)
More from Infra
- Lambda raises $1B debt to buy Nvidia chips for Microsoft — himanshustwts · 2026-08-29
- RX 9070 XT shows huge speed variations with ComfyUI — bosox62 · 2026-08-29
- Prediction: Closed frontier models to become downloadable by 2027 — imjustnewatai · 2026-08-29
- Achieving 181 tok/s on Qwen3.8 with 2x DGX Sparks via NVMe offloading — StartupTim · 2026-08-29
- Together AI processes 135B+ GLM-5.3 Flash tokens in 24 hours — togethercompute · 2026-08-29
- Woof introduces liquidity to AI compute via onchain securitization — edgarpavlovsky · 2026-08-29